[2024-04-22 16:22:58.715882] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=89] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10011, mem_used=76950528) [2024-04-22 16:22:58.716756] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=52] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:22:58.717064] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=37] current monitor number(seq_=-1) [2024-04-22 16:22:58.717171] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:58.717490] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=80] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:58.717582] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=42] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:58.717822] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=55] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:58.724744] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-22 16:22:58.725512] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=17][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:22:58.725546] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-04-22 16:22:58.725562] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-04-22 16:22:58.725571] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=8][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-04-22 16:22:58.725580] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-04-22 16:22:58.725588] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:22:58.725605] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=16][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:22:58.725612] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-04-22 16:22:58.725617] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:22:58.725622] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:22:58.725630] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=8][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:22:58.725635] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:22:58.725640] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:22:58.725645] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:22:58.725650] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:22:58.725662] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:22:58.725667] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:22:58.725673] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:22:58.725678] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:22:58.725688] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:22:58.725695] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:22:58.725701] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:22:58.725713] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=8][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:22:58.725726] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:22:58.725731] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:22:58.725736] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:22:58.725765] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0D-0-0] [lt=4][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:22:58.726322] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=67] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:22:58.727176] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:22:58.727198] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:22:58.727206] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:22:58.727215] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:22:58.727809] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC3F-0-0] [lt=8][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:22:58.727831] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC3F-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:22:58.727841] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC3F-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:22:58.729939] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:131) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] ====== checkpoint timer task ====== [2024-04-22 16:22:58.733169] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1426) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=49] succ to get rec scn(*this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:22:58.733265] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=93] tx ctx memtable get rec scn(this={ObITable:{this:0x7f3cd10be080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f3cd10be080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:22:58.733325] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:856) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=51] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-22 16:22:58.733425] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=33] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:5, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:22:58.733669] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:599) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=237] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1}) [2024-04-22 16:22:58.733711] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:470) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=31] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2024-04-22 16:22:58.733777] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=31] get rec log scn(service_type_=0, rec_log_ts={val:1713774170544377163, v:0}) [2024-04-22 16:22:58.733798] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-22 16:22:58.733820] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=18] get rec log scn(service_type_=2, rec_log_ts={val:1713773717210958135, v:0}) [2024-04-22 16:22:58.733849] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:164) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713773717210958135, v:0}, checkpoint_scn_in_ls_meta={val:1713773717210958135, v:0}, ls_id={id:1}, service_type="DAS_ID") [2024-04-22 16:22:58.733894] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1579) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-22 16:22:58.733936] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:185) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=39] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-22 16:22:58.733957] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:192) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=16] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-22 16:22:58.761968] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1918) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=47] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-22 16:22:58.761994] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1459) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=23] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:22:58.762007] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1476) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:22:58.762030] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2682) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=21] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:22:58.762648] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713774177680355699, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=1082290, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1713774178762645, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-22 16:22:58.762698] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:22:58.762714] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774178762693) [2024-04-22 16:22:58.762726] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774178662635, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:22:58.765763] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2401) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=8] Dumping All Cache Objs(alloc_obj_list.count()=6, alloc_obj_list=[{obj_id:99, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175160}, {obj_id:100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174200}, {obj_id:101, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173640}, {obj_id:102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173992}, {obj_id:103, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:176531}, {obj_id:104, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:175513}]) [2024-04-22 16:22:58.765820] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2690) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=53] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:22:58.769833] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=75] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20719665152, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:22:58.778102] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=93] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=150, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-22 16:22:58.786662] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED78-0-0] [lt=9][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:22:58.786698] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED78-0-0] [lt=25] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:22:58.786714] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED78-0-0] [lt=13][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:22:58.792189] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:113, tid:10899}]) [2024-04-22 16:22:58.820871] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFF9A1-0-0] [lt=118][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774153714643}) [2024-04-22 16:22:58.842250] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:22:58.862411] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=198] tx gc loop thread is running(MTL_ID()=1) [2024-04-22 16:22:58.862462] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=50] try gc retain ctx [2024-04-22 16:22:58.862482] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:748) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] get_max_applied_scn(scn={val:1713774148155695009, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:5, palf_committed_end_lsn_:{lsn:338140}, palf_committed_end_scn_:{val:1713774148155695009, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713774148155695009, v:0}}) [2024-04-22 16:22:58.862513] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1023) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info(lsn={lsn:356459}, scn={val:1713774177680355700, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774177887718, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:356459}, next_to_submit_scn_:{val:1713774177680355700, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:356337}, end_lsn:{lsn:356459}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774177680355699, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774177680355699, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:588032562, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774177680355699, v:0}}}}}) [2024-04-22 16:22:58.862610] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=44] LogLoopThread round_cost_time(us)(round_cost_time=17) [2024-04-22 16:22:58.862760] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1713774177680355699, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1713774178862717}, version_delta=1182404) [2024-04-22 16:22:58.863978] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:253) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=71] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:5, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:22:58.876477] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876502] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876524] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876530] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876536] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876541] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876548] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876553] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876558] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876563] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876570] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876580] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876588] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876594] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876599] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.876604] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:58.891935] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=73] PNIO [ratelimit] time: 1713774178891932, bytes: 65536238, bw: 0.005257 MB/s, add_ts: 1007600, add_bytes: 5554 [2024-04-22 16:22:58.898959] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=83] PNIO [ratelimit] time: 1713774178898957, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007571, add_bytes: 0 [2024-04-22 16:22:58.962852] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:22:58.962908] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:22:58.962947] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774178962840) [2024-04-22 16:22:58.962990] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774178762737, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:22:58.970185] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=62] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20719665152, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:22:58.980907] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=10673, clean_start_pos=3397383, clean_num=125829) [2024-04-22 16:22:59.053561] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] Cache replace map node details(ret=0, replace_node_count=0, replace_time=11173, replace_start_pos=629140, replace_num=62914) [2024-04-22 16:22:59.053625] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=59] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:22:59.154444] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=127] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:22:59.162179] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=28] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:22:59.163016] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:22:59.163060] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774179163009) [2024-04-22 16:22:59.163092] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774178963013, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:22:59.163092] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFB6C-0-0] [lt=40][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713774179163047}) [2024-04-22 16:22:59.181215] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20719665152, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:22:59.210660] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=68][errcode=-4018] try free send task failed(ret=-4018) [2024-04-22 16:22:59.221639] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [10825][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=63] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-22 16:22:59.253737] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:22:59.262993] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=36] ====== tenant freeze timer task ====== [2024-04-22 16:22:59.263078] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:22:59.276943] WDIAG [STORAGE.TRANS] on_timeout (ob_gts_rpc.h:168) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=61][errcode=0] gts rpc timeout(dst="172.23.88.130:2882", tenant_id=1) [2024-04-22 16:22:59.276992] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774178778062]}) [2024-04-22 16:22:59.278668] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:22:59.296762] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4012] sync rpc execute fail(ret=-4012, addr="172.23.88.130:2882", pcode=1315, timeout=1000000) [2024-04-22 16:22:59.296838] WDIAG [SQL.ENG] query_statistics (ob_px_tenant_target_monitor.cpp:275) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4012] send rpc to query statistics failed, need send refresh all(ret=-4012) [2024-04-22 16:22:59.296859] WDIAG [SQL.ENG] refresh_statistics (ob_px_tenant_target_monitor.cpp:114) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4012] query statistics failed(ret=-4012) [2024-04-22 16:22:59.296880] WDIAG [COMMON] operator() (ob_px_target_mgr.h:158) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4012] target monitor refresh statistics failed(ret=-4012, px_tenant_info={v:1}, px_res_info->get_target_monitor()={is_init:true, tenant_id:1, server:"172.23.88.129:2882", dummy_cache_leader:"172.23.88.130:2882", role:2}) [2024-04-22 16:22:59.304885] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=0][errcode=-4012] sync rpc execute fail(ret=-4012, addr="172.23.88.130:2882", pcode=1282, timeout=1300651) [2024-04-22 16:22:59.304919] WDIAG [SQL.EXE] task_execute (ob_executor_rpc_impl.cpp:81) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=26][errcode=-4012] rpc task_execute fail(ret=-4012, tenant_id=1, svr="172.23.88.130:2882", timeout=1300651, timeout_timestamp=1713774179310629) [2024-04-22 16:22:59.304956] WDIAG [SQL.EXE] execute (ob_remote_task_executor.cpp:99) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=9][errcode=-4012] fail post task(ret=-4012) [2024-04-22 16:22:59.304975] WDIAG [SQL.EXE] execute (ob_remote_job_executor.cpp:55) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=12][errcode=-4012] fail execute task(ret=-4012, *task_info={task_loc:{server:"172.23.88.130:2882", ob_task_id:{ob_job_id:{ob_execution_id:{server:"172.23.88.129:2882", execution_id:18446744073709551615, task_type:0, hash:4219408603045787357}, job_id:19077}, task_id:0, task_cnt:0}}, range_location:{part_locs:[cnt:0], server:"0.0.0.0:0"}, location_idx:0, location_idx_list:[cnt:1, 0], state:5, slice_count_pos:[cnt:0], background:false, retry_times:0, location_idx_list:[cnt:1, 0]}) [2024-04-22 16:22:59.305021] WDIAG [SQL.EXE] execute_with_plan (ob_remote_scheduler.cpp:124) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=45][errcode=-4012] fail execute remote job(ret=-4012) [2024-04-22 16:22:59.305042] WDIAG [SQL.EXE] execute_with_plan (ob_remote_scheduler.cpp:141) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=7][errcode=-4012] fail to schedule, print job's status(ret=-4012, print_ret=0, job_status="{job_count:2, {task_count:0, state statistics:{}}, {task_count:1, task_info:{loc:"172.23.88.130:2882"ctrl:"172.23.88.129:2882"eid:18446744073709551615jid:19077tid:0pull_sid:0state:5}state statistics:{failed:1}}}") [2024-04-22 16:22:59.305056] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:57) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=9][errcode=-4012] execute with plan failed(ret=-4012) [2024-04-22 16:22:59.305075] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=14][errcode=-4012] fail execute plan(ret=-4012) [2024-04-22 16:22:59.305083] WDIAG [SQL] open (ob_result_set.cpp:158) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=6][errcode=-4012] execute plan failed(ret=-4012) [2024-04-22 16:22:59.305088] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=4][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:22:59.305095] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=6][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:22:59.305105] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=8][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=34, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:22:59.305119] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=12][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:34, local_retry_times:34, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:22:59.305143] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=15][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:22:59.305148] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=4][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:22:59.305153] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=5][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:22:59.305165] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC09-0-0] [lt=5][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:22:59.305173] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=6] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1993710) [2024-04-22 16:22:59.305205] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=31][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:22:59.305217] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=9][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:22:59.305224] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=6][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:22:59.305231] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=6][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:22:59.305237] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=5][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:22:59.305252] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=12][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:22:59.305328] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=8][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:22:59.305342] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=13][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:22:59.305350] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=8][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:22:59.305358] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=7][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:22:59.305371] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=12][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:22:59.305379] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:521) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=7][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-04-22 16:22:59.305387] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:236) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC09-0-0] [lt=6][errcode=-4012] fail to process refresh task(ret=-4012, ret="OB_TIMEOUT", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-04-22 16:22:59.305407] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:599) [10476][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4012] fail to batch process task(ret=-4012) [2024-04-22 16:22:59.305413] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:499) [10476][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4012] fail to batch execute task(ret=-4012, tasks.count()=1) [2024-04-22 16:22:59.305452] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:261) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=17] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:22:59.305471] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:305) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=16] [REFRESH_SCHEMA] begin refresh schema, (begin_time=1713774179305471) [2024-04-22 16:22:59.305478] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2388) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[]) [2024-04-22 16:22:59.310790] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=50] ObArchiveSender is running(thread_index=0) [2024-04-22 16:22:59.323132] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.363216] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:22:59.363242] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774179363207) [2024-04-22 16:22:59.363253] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774179163107, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:22:59.381576] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20721762304, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:22:59.453885] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:22:59.463283] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:22:59.463327] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774179463278) [2024-04-22 16:22:59.463339] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774179363261, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:22:59.563418] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:22:59.563546] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=124][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774179563403) [2024-04-22 16:22:59.563643] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=94][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774179463348, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:22:59.581908] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20721762304, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:22:59.582188] INFO [COMMON] wash (ob_kvcache_store.cpp:356) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash time detail, (compute_wash_size_time=106, refresh_score_time=42, wash_time=0) [2024-04-22 16:22:59.654049] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:22:59.706979] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.707236] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.707593] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.707651] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.707715] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.707937] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.708163] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.708503] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.708638] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.708679] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.709602] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.712151] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.712391] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.712482] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.712591] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.712593] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2024-04-22 16:22:59.712614] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=15] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:22:59.712636] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=10] cannot_recycle_log_size statistics(cannot_recycle_log_size=356459, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:22:59.712728] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [10543][Blacklist][T0][Y0-0000000000000000-0-0] [lt=59] blacklist_loop exec finished(cost_time=48, is_enabled=true, send_cnt=0) [2024-04-22 16:22:59.713536] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.713711] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.713824] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.714781] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=56] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.714826] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=38] gc stale ls task succ [2024-04-22 16:22:59.716046] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=55] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=10016, mem_used=76950528) [2024-04-22 16:22:59.716851] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:22:59.717186] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=16] current monitor number(seq_=-1) [2024-04-22 16:22:59.717458] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=46] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.717636] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.717786] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.718259] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:22:59.726515] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:22:59.727455] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:22:59.727478] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:22:59.727491] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:22:59.727504] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:22:59.763595] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:22:59.763640] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774179763589) [2024-04-22 16:22:59.763659] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774179563687, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:22:59.771810] WDIAG [STORAGE.TRANS] on_timeout (ob_gts_rpc.h:168) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=59][errcode=0] gts rpc timeout(dst="172.23.88.130:2882", tenant_id=1) [2024-04-22 16:22:59.771847] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774179278627]}) [2024-04-22 16:22:59.782819] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20721762304, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:22:59.787699] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:22:59.787748] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=37] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:22:59.787765] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=14][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:22:59.787774] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=9][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:22:59.787782] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=7][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:22:59.787792] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=7][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:22:59.787799] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:22:59.787813] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:22:59.787818] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:22:59.787822] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:22:59.787829] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:22:59.787833] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:22:59.787846] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:22:59.787852] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:22:59.787857] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:22:59.787869] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=6][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:22:59.787875] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:22:59.787882] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=6][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:22:59.787887] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:22:59.787894] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:22:59.787904] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=9][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:22:59.787911] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:22:59.787927] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=12][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:22:59.787939] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:22:59.787948] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:22:59.787955] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:22:59.787974] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED79-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:22:59.787982] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=6][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:22:59.787989] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:22:59.787995] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:22:59.788001] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:22:59.788007] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=6][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd227a050, start=1713774179787554, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:22:59.788014] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=7][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:22:59.788019] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=4][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:22:59.788082] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=14][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:22:59.788089] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED79-0-0] [lt=6][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:22:59.821391] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10483][LSSysTblUp0][T0][YB42AC175881-000616AB0D0FF9C9-0-0] [lt=52][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774153714643}) [2024-04-22 16:22:59.854178] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:22:59.877544] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877603] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877631] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877788] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=148] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877821] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877832] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877841] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877851] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877859] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877868] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877876] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877885] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877893] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877961] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=66] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877977] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.877988] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:22:59.899525] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO [ratelimit] time: 1713774179899523, bytes: 65536238, bw: 0.000000 MB/s, add_ts: 1007591, add_bytes: 0 [2024-04-22 16:22:59.906563] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1713774179906561, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007604, add_bytes: 0 [2024-04-22 16:22:59.963771] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:22:59.963811] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2107) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=75] do_fetch_log_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", dest="172.23.88.130:2882", fetch_log_count=2048, fetch_start_lsn={lsn:356459}, prev_lsn={lsn:356337}, fetch_start_log_id=2925, last_slide_log_id=2924, fetch_log_size=41943040, accepted_mode_pid=6, last_fetch_req_time=1713774179963810, last_fetch_end_lsn={lsn:42299499}, last_fetch_max_log_id=4972, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.23.88.129:2882", lsn_allocator:{max_log_id:2925, max_lsn:356581, max_scn:{val:1713774177880657607, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:356581}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:356459}, last_submit_end_lsn:{lsn:356581}, last_submit_log_id:2925, last_submit_log_pid:6, max_flushed_lsn:{lsn:356459}, max_flushed_end_lsn:{lsn:356581}, max_flushed_log_pid:6, committed_end_lsn:{lsn:356459}, last_slide_log_id:2924, last_slide_scn:{val:1713774177680355699, v:0}, last_slide_lsn:{lsn:356337}, last_slide_end_lsn:{lsn:356459}, last_slide_log_pid:6, last_slide_log_accum_checksum:588032562, last_fetch_end_lsn:{lsn:42299499}, last_fetch_max_log_id:4972, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713774179963810, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f3cd2de4530}) [2024-04-22 16:22:59.963834] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:22:59.963884] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774179963754) [2024-04-22 16:22:59.963943] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774179763681, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:22:59.983527] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=69] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20719665152, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:00.034977] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=41] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:00.054308] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:00.163054] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=159] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:00.163972] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:00.164039] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774180163960) [2024-04-22 16:23:00.164084] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774179963964, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:00.183879] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=66] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20719665152, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:00.227179] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4012] sync rpc execute fail(ret=-4012, addr="172.23.88.130:2882", pcode=5407, timeout=2000000) [2024-04-22 16:23:00.227222] WDIAG [CLOG] post_sync_request_to_server_ (log_net_service.h:347) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4012] ObLogRpc post_sync_request failed(ret=-4012, palf_id=1, req={get_type:1}, server="172.23.88.130:2882") [2024-04-22 16:23:00.227235] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5195) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] get_palf_max_scn failed(ret=-4012, palf_id=1) [2024-04-22 16:23:00.227260] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5155) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] get_palf_max_scn failed(ret=-4012, self="172.23.88.129:2882", palf_id=1) [2024-04-22 16:23:00.227268] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5173) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=8] is_in_sync(ret=-4012, palf_id=1, is_in_sync=false, is_use_cache=false, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1713774177880657607, v:0}) [2024-04-22 16:23:00.227292] WDIAG [PALF] runTimerTask (log_updater.cpp:109) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] update_palf cost too much time(ret=0, cost_time_us=2003310, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-04-22 16:23:00.227310] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFB70-0-0] [lt=24][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713774180227245}) [2024-04-22 16:23:00.230721] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=102][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:00.230869] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=8][errcode=0] execute rpc failed(rc=-4012, server="172.23.88.130:2882", timeout=2000000, packet code=330, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}) [2024-04-22 16:23:00.231020] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=124][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:00.231079] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=26][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:00.231261] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=176][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:00.231359] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=95] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:00.231522] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=156] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2024-04-22 16:23:00.233660] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=23][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:00.233735] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:00.233802] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=61][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:00.233822] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:00.233841] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:00.233860] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:00.233874] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:00.233892] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:00.233906] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:00.233938] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=30][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:00.233950] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:00.233962] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:00.233975] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:00.233987] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:00.233999] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:00.234026] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:00.234041] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.234056] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.234069] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:00.234081] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:00.234096] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:00.234110] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=12][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:00.234148] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=29][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:00.234178] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:00.234191] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=12][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:00.234201] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:00.234226] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:00.234243] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:00.234258] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:00.234280] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=20][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:00.234300] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=18][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:00.234321] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=19][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45f0050, start=1713774180233363, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:00.234363] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=40][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:00.234383] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:00.234543] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=24][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:00.234587] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=40][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:00.234612] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=22][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:00.234628] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=14][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:00.234643] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=11][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:00.234661] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=14][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:00.234673] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED75-0-0] [lt=12][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:00.254448] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:00.264052] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:00.279534] WDIAG [STORAGE.TRANS] on_timeout (ob_gts_rpc.h:168) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] gts rpc timeout(dst="172.23.88.130:2882", tenant_id=1) [2024-04-22 16:23:00.279582] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774179779160]}) [2024-04-22 16:23:00.279706] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:00.322270] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.364121] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:00.364165] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774180364114) [2024-04-22 16:23:00.364181] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774180164122, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:00.384208] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20721762304, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:00.454566] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:00.464235] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:00.464300] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774180464222) [2024-04-22 16:23:00.464340] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774180364189, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:00.584569] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20721762304, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:00.654664] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:00.664379] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:00.664409] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774180664373) [2024-04-22 16:23:00.664431] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774180464391, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:00.706792] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=78] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.706965] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.707263] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.707768] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.707827] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.707946] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.707961] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.708199] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.708745] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.709253] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.709827] INFO pn_ratelimit (group.c:589) [10608][IngressService][T0][Y0-0000000000000000-0-0] [lt=47] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:23:00.710713] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.711394] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=15][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:00.711489] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=71] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:00.711542] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=49][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:23:00.711557] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:00.711582] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=23][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:23:00.711597] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:00.711606] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:00.711620] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:23:00.711630] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=8][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:00.711638] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:00.711647] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=8][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:00.711670] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=22][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:23:00.711679] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=8][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:23:00.711687] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=7][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:23:00.711696] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:00.711705] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:00.711722] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=15][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:00.711731] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:00.711753] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:00.711765] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.711778] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.711788] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:00.711798] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:23:00.711810] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:00.711820] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:00.711828] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.711845] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=18][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:00.711885] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=31][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:00.711899] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:00.711911] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=11][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:00.711976] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4D-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:00.712002] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10899][BlackListServic][T0][YB42AC175881-000616AB07BFEC4D-0-0] [lt=21][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:23:00.712019] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:00.712036] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:00.712049] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:00.712068] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1460050, start=1713774180710988, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:00.712085] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:00.712095] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:00.712198] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12] ls blacklist refresh finish(cost_time=2377) [2024-04-22 16:23:00.712247] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=103] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.712588] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.713193] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.713584] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.713648] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.713916] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.714383] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.714940] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=11] gc stale ls task succ [2024-04-22 16:23:00.716149] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10019, mem_used=76950528) [2024-04-22 16:23:00.716952] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:00.716990] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.717342] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=20] current monitor number(seq_=-1) [2024-04-22 16:23:00.717486] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.717535] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.718332] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:00.720660] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=10][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:00.720734] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:00.720784] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=44][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:00.720822] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=35][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:00.720845] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=21][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:00.720870] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=18][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:00.720907] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=35][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:00.720933] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=17][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:00.720947] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=12][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:00.720959] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=11][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:00.720971] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=11][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:00.721109] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=130][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:00.721152] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=38][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:00.721194] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=23][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:00.721212] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.721234] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=17][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.721254] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=17][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:00.721269] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:00.721282] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:00.721291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:00.721312] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:00.721381] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=61][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:00.721393] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:00.721400] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:00.721417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10480][Occam][T1][YB42AC175881-000616AB083FEC5D-0-0] [lt=6][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:00.721461] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10480][Occam][T0][YB42AC175881-000616AB083FEC5D-0-0] [lt=40][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:00.721484] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882)) [2024-04-22 16:23:00.721524] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:00.721543] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:00.721588] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882)") [2024-04-22 16:23:00.721602] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3ceeb84050, start=1713774180718669, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882)) [2024-04-22 16:23:00.721660] WDIAG [SHARE] operator() (ob_event_history_table_operator.cpp:393) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] sync execute sql failed(sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774058333277), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:8', '172.23.88.129', 2882)", ret=-5019) [2024-04-22 16:23:00.721902] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:00.721949] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0A-0-0] [lt=197][errcode=0] execute rpc failed(rc=-4012, server="172.23.88.130:2882", timeout=2000000, packet code=330, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}) [2024-04-22 16:23:00.722002] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0A-0-0] [lt=32][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:00.722049] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0A-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:00.722128] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0A-0-0] [lt=76][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:00.722164] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0A-0-0] [lt=35] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:00.722195] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0A-0-0] [lt=28] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2024-04-22 16:23:00.722221] INFO [SHARE.LOCATION] merge_leader_from (ob_location_struct.cpp:488) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0A-0-0] [lt=8] [LS_LOCATION] leader will be changed(new_leader={server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}, old_leader=NULL, exist_replica=NULL, old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774180722215, replica_locations:[{server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}, {server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}]}) [2024-04-22 16:23:00.722264] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0A-0-0] [lt=39] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774173722401, replica_locations:[{server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}, {server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774180722215, replica_locations:[{server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}]}) [2024-04-22 16:23:00.722575] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:00.722646] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2024-04-22 16:23:00.722754] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=103][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2024-04-22 16:23:00.722810] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=53][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2024-04-22 16:23:00.722834] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=22][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2024-04-22 16:23:00.722860] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=18][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:00.722890] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=28][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:00.722909] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2024-04-22 16:23:00.722922] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:00.722942] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=18][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:00.722956] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=13][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:00.722967] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:00.722979] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:00.722990] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:00.723002] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:00.723035] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=20][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:00.723049] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.723065] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.723077] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:00.723075] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=18][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:00.723088] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", ret=-5019) [2024-04-22 16:23:00.723104] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:00.723110] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=21] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:00.723117] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:00.723130] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=17][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:00.723144] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:00.723144] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:00.723172] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:00.723183] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:00.723193] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:00.723157] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:00.723212] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=50][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:00.723221] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:00.723216] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:00.723234] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:00.723243] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:00.723251] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=7][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:00.723259] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=8][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:00.723267] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=7][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:00.723275] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=6][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:00.723272] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD5-0-0] [lt=23][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, aret=-5019, ret=-5019) [2024-04-22 16:23:00.723291] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:00.723294] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:00.723304] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.723310] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:00.723318] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.723322] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:00.723337] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd2206050, start=1713774180722236, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:00.723354] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:00.723331] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:00.723368] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:943) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:00.723376] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10480][Occam][T1][YB42AC175881-000616AB083FEC5E-0-0] [lt=41][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:00.723554] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:672) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:00.726692] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=64] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:00.727741] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:00.727765] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:00.727773] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:00.727794] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:00.751547] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:00.751654] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED69-0-0] [lt=82][errcode=0] execute rpc failed(rc=-4012, server="172.23.88.130:2882", timeout=2000000, packet code=1193, arg={addr:"172.23.88.129:2882"}) [2024-04-22 16:23:00.751693] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:875) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED69-0-0] [lt=27][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:00.752236] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A373238-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:00.752279] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A373238-0-0] [lt=40][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:00.752298] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A373238-0-0] [lt=18][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:00.752312] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A373238-0-0] [lt=13][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:00.752858] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10918][T1_L0_G2][T1][YB42AC175883-000616AB0A373238-0-0] [lt=82][errcode=0] compare between invalid priority [2024-04-22 16:23:00.764478] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=3083777, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:00.764508] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=35, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:00.780243] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774180780206]}) [2024-04-22 16:23:00.784265] WDIAG [STORAGE.TRANS] on_timeout (ob_gts_rpc.h:168) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=50][errcode=0] gts rpc timeout(dst="172.23.88.130:2882", tenant_id=1) [2024-04-22 16:23:00.784315] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774180780206]}) [2024-04-22 16:23:00.784887] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20719665152, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:00.788889] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=7][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:00.788923] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=25] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:00.788942] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=18][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:00.788953] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:00.788962] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=9][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:00.788978] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:00.788983] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:00.788989] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:00.788994] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:00.789009] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=14][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:00.789014] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:00.789019] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:00.789026] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=6][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:00.789032] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:00.789038] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:00.789049] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:00.789059] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:00.789066] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7A-0-0] [lt=6][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:00.795464] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:39, tid:10410}]) [2024-04-22 16:23:00.797197] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4012] sync rpc execute fail(ret=-4012, addr="172.23.88.130:2882", pcode=1315, timeout=1000000) [2024-04-22 16:23:00.797229] WDIAG [SQL.ENG] query_statistics (ob_px_tenant_target_monitor.cpp:275) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4012] send rpc to query statistics failed, need send refresh all(ret=-4012) [2024-04-22 16:23:00.797347] WDIAG [SQL.ENG] refresh_statistics (ob_px_tenant_target_monitor.cpp:114) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=110][errcode=-4012] query statistics failed(ret=-4012) [2024-04-22 16:23:00.797372] WDIAG [COMMON] operator() (ob_px_target_mgr.h:158) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4012] target monitor refresh statistics failed(ret=-4012, px_tenant_info={v:1}, px_res_info->get_target_monitor()={is_init:true, tenant_id:1, server:"172.23.88.129:2882", dummy_cache_leader:"172.23.88.130:2882", role:2}) [2024-04-22 16:23:00.823495] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFA67-0-0] [lt=33][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774153714643}) [2024-04-22 16:23:00.828706] INFO pn_print_diag_info (group.c:675) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO client:0x7f3d01bfa048_172.23.88.129:52390_172.23.88.130:2882_201_1713774058940221_1, write_queue=0/0, write=367/522252, read=363/91707, doing=1, done=366, write_time=408504, read_time=111, process_time=179 [2024-04-22 16:23:00.828754] INFO pn_print_diag_info (group.c:675) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=42] PNIO client:0x7f3d01a77388_172.23.88.129:33086_172.23.88.129:2882_228_1713774039830173_1, write_queue=0/0, write=127/45437, read=127/30669, doing=0, done=127, write_time=135096, read_time=26, process_time=47 [2024-04-22 16:23:00.828784] INFO pn_print_diag_info (group.c:675) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=23] PNIO client:0x7f3d01a769e8_172.23.88.129:59664_172.23.88.131:2882_225_1713774039818052_1, write_queue=0/0, write=219/68252, read=219/48356, doing=0, done=219, write_time=248973, read_time=59, process_time=102 [2024-04-22 16:23:00.828826] INFO pn_print_diag_info (group.c:687) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO server:0x7f3d01ba49e8_172.23.88.131:36178_206_1713774154108616, write_queue=0/0, write=104/40222, read=104/27383, doing=0, done=104, write_time=116663, read_time=31, process_time=51 [2024-04-22 16:23:00.828846] INFO pn_print_diag_info (group.c:687) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO server:0x7f3cd24d9358_172.23.88.130:55268_197_1713774154028168, write_queue=0/0, write=49/10314, read=49/16706, doing=0, done=49, write_time=52342, read_time=13, process_time=77 [2024-04-22 16:23:00.828857] INFO pn_print_diag_info (group.c:687) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO server:0x7f3cd24d8048_172.23.88.129:33086_230_1713774039831518, write_queue=0/0, write=127/30669, read=127/45437, doing=0, done=127, write_time=133810, read_time=67, process_time=105 [2024-04-22 16:23:00.828867] INFO pn_print_diag_info (group.c:694) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO client_send:3301/5326544, client_queue_time=1851159, cnt=3, server_send:6427/2179636, server_queue_time=475051, cnt=3 [2024-04-22 16:23:00.845130] INFO pn_print_diag_info (group.c:675) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO client:0x7f3cd33fe048_172.23.88.129:52384_172.23.88.130:2882_195_1713774058872676_1, write_queue=0/0, write=361/486582, read=358/88477, doing=2, done=359, write_time=398699, read_time=88, process_time=154 [2024-04-22 16:23:00.845157] INFO pn_print_diag_info (group.c:675) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO client:0x7f3cd33f8048_172.23.88.129:59662_172.23.88.131:2882_224_1713774038824972_1, write_queue=0/0, write=207/67580, read=207/45182, doing=0, done=207, write_time=226001, read_time=55, process_time=96 [2024-04-22 16:23:00.845171] INFO pn_print_diag_info (group.c:675) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO client:0x7f3cd3204048_172.23.88.129:32940_172.23.88.129:2882_196_1713773677281877_1, write_queue=0/0, write=356/117323, read=356/89357, doing=0, done=356, write_time=402041, read_time=116, process_time=181 [2024-04-22 16:23:00.845177] INFO pn_print_diag_info (group.c:687) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x7f3cd33fe9e8_172.23.88.130:55272_212_1713774154328340, write_queue=0/0, write=49/10434, read=49/17661, doing=0, done=49, write_time=53317, read_time=11, process_time=65 [2024-04-22 16:23:00.845186] INFO pn_print_diag_info (group.c:687) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO server:0x7f3cd33f6048_172.23.88.131:36176_205_1713774154105635, write_queue=0/0, write=95/37176, read=95/25158, doing=0, done=95, write_time=113266, read_time=32, process_time=86 [2024-04-22 16:23:00.845192] INFO pn_print_diag_info (group.c:687) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x7f3cd32049e8_172.23.88.129:32940_199_1713773677282496, write_queue=0/0, write=356/89357, read=356/117323, doing=0, done=356, write_time=392110, read_time=70, process_time=134 [2024-04-22 16:23:00.845197] INFO pn_print_diag_info (group.c:694) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO client_send:3198/5397312, client_queue_time=4481404, cnt=3, server_send:6162/2087024, server_queue_time=4307365, cnt=3 [2024-04-22 16:23:00.854817] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:00.857988] INFO pn_print_diag_info (group.c:675) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO client:0x7f3cd332d358_172.23.88.129:59696_172.23.88.131:2882_229_1713774159723000_1, write_queue=0/0, write=7/1635, read=7/2433, doing=0, done=7, write_time=7923, read_time=2, process_time=3 [2024-04-22 16:23:00.858010] INFO pn_print_diag_info (group.c:675) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO client:0x7f3cd332c048_172.23.88.129:33114_172.23.88.129:2882_218_1713774156719882_1, write_queue=0/0, write=8/1855, read=8/2349, doing=0, done=8, write_time=11069, read_time=2, process_time=2 [2024-04-22 16:23:00.858019] INFO pn_print_diag_info (group.c:675) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x7f3cd33d4048_172.23.88.129:52380_172.23.88.130:2882_188_1713774058833536_1, write_queue=0/0, write=384/508028, read=381/96087, doing=2, done=382, write_time=418541, read_time=102, process_time=176 [2024-04-22 16:23:00.858029] INFO pn_print_diag_info (group.c:687) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO server:0x7f3cd332c9e8_172.23.88.129:33114_219_1713774156721290, write_queue=0/0, write=8/2349, read=8/1855, doing=0, done=8, write_time=9559, read_time=3, process_time=4 [2024-04-22 16:23:00.858038] INFO pn_print_diag_info (group.c:687) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO server:0x7f3cd1aad328_172.23.88.130:55280_217_1713774155737928, write_queue=0/0, write=51/10806, read=51/15769, doing=0, done=51, write_time=51685, read_time=9, process_time=23 [2024-04-22 16:23:00.858051] INFO pn_print_diag_info (group.c:687) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO server:0x7f3cd33f89e8_172.23.88.131:36174_202_1713774154102081, write_queue=0/0, write=93/35595, read=93/24564, doing=0, done=93, write_time=100368, read_time=32, process_time=49 [2024-04-22 16:23:00.858055] INFO pn_print_diag_info (group.c:694) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO client_send:3288/5422375, client_queue_time=2118200, cnt=3, server_send:6254/2125612, server_queue_time=437037, cnt=3 [2024-04-22 16:23:00.864536] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:00.864582] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774180864530) [2024-04-22 16:23:00.864602] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774180664453, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:00.864640] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=3183513, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:00.864659] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:00.866129] INFO pn_print_diag_info (group.c:675) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=54] PNIO client:0x7f3d01be69b8_172.23.88.129:52392_172.23.88.130:2882_208_1713774059033687_1, write_queue=0/0, write=380/512878, read=378/95319, doing=1, done=379, write_time=423123, read_time=110, process_time=189 [2024-04-22 16:23:00.866154] INFO pn_print_diag_info (group.c:675) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO client:0x7f3d01a76048_172.23.88.129:59658_172.23.88.131:2882_220_1713774035812683_1, write_queue=0/0, write=199/63544, read=199/44243, doing=0, done=199, write_time=229932, read_time=50, process_time=107 [2024-04-22 16:23:00.866174] INFO pn_print_diag_info (group.c:675) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO client:0x7f3d01ba5388_172.23.88.129:32960_172.23.88.129:2882_209_1713773678163497_1, write_queue=0/0, write=339/114629, read=339/84601, doing=0, done=339, write_time=377511, read_time=110, process_time=196 [2024-04-22 16:23:00.866181] INFO pn_print_diag_info (group.c:687) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x7f3d01bf8048_172.23.88.131:36186_193_1713774155164603, write_queue=0/0, write=86/32661, read=86/23397, doing=0, done=86, write_time=96515, read_time=23, process_time=40 [2024-04-22 16:23:00.866193] INFO pn_print_diag_info (group.c:687) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO server:0x7f3d01a05078_172.23.88.130:55276_192_1713774155034559, write_queue=0/0, write=46/9732, read=46/15019, doing=0, done=46, write_time=51376, read_time=16, process_time=25 [2024-04-22 16:23:00.866198] INFO pn_print_diag_info (group.c:687) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO server:0x7f3d01be6048_172.23.88.129:32960_210_1713773678164478, write_queue=0/0, write=339/84601, read=339/114629, doing=0, done=339, write_time=366879, read_time=73, process_time=123 [2024-04-22 16:23:00.866202] INFO pn_print_diag_info (group.c:694) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO client_send:3204/5363355, client_queue_time=1748151, cnt=3, server_send:6090/2078283, server_queue_time=419037, cnt=3 [2024-04-22 16:23:00.870908] INFO pn_print_diag_info (group.c:675) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=64] PNIO client:0x7f3cd24d89b8_172.23.88.129:59690_172.23.88.131:2882_203_1713774153714701_1, write_queue=0/0, write=9/2090, read=9/2742, doing=0, done=9, write_time=9865, read_time=1, process_time=4 [2024-04-22 16:23:00.870926] INFO pn_print_diag_info (group.c:675) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO client:0x7f3d01ba4048_172.23.88.129:52388_172.23.88.130:2882_198_1713774058933621_1, write_queue=0/0, write=381/498424, read=379/96669, doing=1, done=380, write_time=412515, read_time=102, process_time=179 [2024-04-22 16:23:00.870935] INFO pn_print_diag_info (group.c:675) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO client:0x7f3d01bfa9e8_172.23.88.129:32980_172.23.88.129:2882_232_1713773680822263_1, write_queue=0/0, write=347/116028, read=347/88221, doing=0, done=347, write_time=394981, read_time=75, process_time=146 [2024-04-22 16:23:00.870941] INFO pn_print_diag_info (group.c:687) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO server:0x7f3d01a04708_172.23.88.131:36184_215_1713774154696068, write_queue=0/0, write=90/34575, read=90/23864, doing=0, done=90, write_time=98136, read_time=29, process_time=41 [2024-04-22 16:23:00.870948] INFO pn_print_diag_info (group.c:687) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x7f3d01be7358_172.23.88.130:55270_211_1713774154239144, write_queue=0/0, write=49/10338, read=49/17462, doing=0, done=49, write_time=52988, read_time=12, process_time=24 [2024-04-22 16:23:00.870963] INFO pn_print_diag_info (group.c:687) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO server:0x7f3d01bfb388_172.23.88.129:32980_233_1713773680822594, write_queue=0/0, write=347/88221, read=347/116028, doing=0, done=347, write_time=373701, read_time=79, process_time=138 [2024-04-22 16:23:00.870971] INFO pn_print_diag_info (group.c:694) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO client_send:3295/5434371, client_queue_time=622059, cnt=3, server_send:6342/2160724, server_queue_time=451044, cnt=3 [2024-04-22 16:23:00.879012] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879044] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879054] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879062] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879068] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879076] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879081] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879087] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879210] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=116] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879234] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879243] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879249] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879256] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879262] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879268] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.879274] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:00.907146] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1713774180907144, bytes: 65538583, bw: 0.002219 MB/s, add_ts: 1007621, add_bytes: 2345 [2024-04-22 16:23:00.907182] INFO pn_print_diag_info (group.c:675) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO client:0x7f3cd3205358_172.23.88.129:52394_172.23.88.130:2882_204_1713774153714251_1, write_queue=0/0, write=179/458130, read=177/53099, doing=0, done=179, write_time=198192, read_time=41, process_time=71 [2024-04-22 16:23:00.907192] INFO pn_print_diag_info (group.c:675) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO client:0x7f3cd33f7328_172.23.88.129:59660_172.23.88.131:2882_221_1713774038815655_1, write_queue=0/0, write=207/66556, read=207/45096, doing=0, done=207, write_time=232364, read_time=65, process_time=106 [2024-04-22 16:23:00.907198] INFO pn_print_diag_info (group.c:675) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO client:0x7f3cd33d49e8_172.23.88.129:32974_172.23.88.129:2882_226_1713773680810120_1, write_queue=0/0, write=390/130281, read=390/99221, doing=0, done=390, write_time=469806, read_time=94, process_time=176 [2024-04-22 16:23:00.907205] INFO pn_print_diag_info (group.c:687) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO server:0x7f3cd33ff358_172.23.88.130:55274_213_1713774154477802, write_queue=0/0, write=55/11789, read=55/18502, doing=0, done=55, write_time=59625, read_time=15, process_time=24 [2024-04-22 16:23:00.907215] INFO pn_print_diag_info (group.c:687) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO server:0x7f3cd33f69b8_172.23.88.131:36180_207_1713774154226935, write_queue=0/0, write=100/38823, read=100/26554, doing=0, done=100, write_time=108645, read_time=29, process_time=46 [2024-04-22 16:23:00.907220] INFO pn_print_diag_info (group.c:687) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO server:0x7f3cd33d5388_172.23.88.129:32974_227_1713773680810566, write_queue=0/0, write=390/99221, read=390/130281, doing=0, done=390, write_time=453512, read_time=83, process_time=155 [2024-04-22 16:23:00.907226] INFO pn_print_diag_info (group.c:694) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO client_send:3293/5421427, client_queue_time=1935179, cnt=3, server_send:6425/2201239, server_queue_time=443047, cnt=3 [2024-04-22 16:23:00.910317] INFO pn_print_diag_info (group.c:675) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO client:0x7f3cd3372048_172.23.88.129:52382_172.23.88.130:2882_189_1713774058839919_1, write_queue=0/0, write=368/506645, read=364/93069, doing=2, done=366, write_time=403884, read_time=105, process_time=172 [2024-04-22 16:23:00.910338] INFO pn_print_diag_info (group.c:675) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO client:0x7f3cd33f9388_172.23.88.129:59670_172.23.88.131:2882_231_1713774040820226_1, write_queue=0/0, write=193/62137, read=193/41559, doing=0, done=193, write_time=216429, read_time=56, process_time=94 [2024-04-22 16:23:00.910358] INFO pn_print_diag_info (group.c:675) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO client:0x7f3cd33729e8_172.23.88.129:32964_172.23.88.129:2882_222_1713773680808184_1, write_queue=0/0, write=375/124235, read=375/94664, doing=0, done=375, write_time=427277, read_time=96, process_time=170 [2024-04-22 16:23:00.910363] INFO pn_print_diag_info (group.c:687) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO server:0x7f3cd1aac9b8_172.23.88.130:55278_216_1713774155481070, write_queue=0/0, write=37/7866, read=37/12026, doing=0, done=37, write_time=38871, read_time=9, process_time=18 [2024-04-22 16:23:00.910368] INFO pn_print_diag_info (group.c:687) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO server:0x7f3cd1aac048_172.23.88.131:36182_214_1713774154577745, write_queue=0/0, write=93/36390, read=93/24787, doing=0, done=93, write_time=101062, read_time=31, process_time=40 [2024-04-22 16:23:00.910373] INFO pn_print_diag_info (group.c:687) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO server:0x7f3cd3373388_172.23.88.129:32964_223_1713773680808410, write_queue=0/0, write=375/94664, read=375/124235, doing=0, done=375, write_time=401515, read_time=76, process_time=136 [2024-04-22 16:23:00.910377] INFO pn_print_diag_info (group.c:694) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO client_send:3195/5275439, client_queue_time=7222618, cnt=3, server_send:6237/2134830, server_queue_time=419034, cnt=3 [2024-04-22 16:23:00.914183] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1713774180914181, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007620, add_bytes: 0 [2024-04-22 16:23:00.914219] INFO pn_print_diag_info (group.c:694) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO client_send:0/0, client_queue_time=0, cnt=0, server_send:0/0, server_queue_time=0, cnt=0 [2024-04-22 16:23:00.915691] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:519) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=875] worker thread began to shrink(tenant_->id()=1, group_id_=19, token=0) [2024-04-22 16:23:00.924291] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:534) [11409][T1_L0_G19][T1][Y0-0000000000000000-0-0] [lt=6] worker thread exit(tenant_->id()=1, workers_.get_size()=1) [2024-04-22 16:23:00.924332] INFO [SHARE] end_run (ob_tenant_base.cpp:332) [11409][T1_L0_G19][T0][Y0-0000000000000000-0-0] [lt=23] tenant thread end_run(id_=1, ret=0, thread_count_=226) [2024-04-22 16:23:00.952406] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:677) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=12] mark black, addr: "172.23.88.130:2882", last_write_ts_=1713774178151242, last_read_ts_=1713774177951202 [2024-04-22 16:23:00.964629] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=3283311, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:00.964677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:00.985207] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20719665152, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:00.997460] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=12221, clean_start_pos=3523212, clean_num=125829) [2024-04-22 16:23:01.054921] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:01.064694] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:01.064733] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:01.064749] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774181064679) [2024-04-22 16:23:01.064764] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774180864609, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:01.064798] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=3384031, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.064829] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.123497] INFO keepalive_check (time_wheel.c:71) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO socket dest server in blacklist, it will be destroyed, sock=(ptr=0x7f3d01bfa048,dest=-2108155988:2882) [2024-04-22 16:23:01.124063] INFO eloop_handle_sock_event (eloop.c:95) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO sock destroy: sock=0x7f3d01bfa048, connection=fd:201:local:172.23.88.129:52390:remote:172.23.88.130:2882, s->mask=0xc [2024-04-22 16:23:01.124127] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO sk_destroy: s=0x7f3d01bfa048 io=0x7f3d00984608 [2024-04-22 16:23:01.124140] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO resp_cb on sk_destroy: packet_id=22768 s=0x7f3d01bfa048 [2024-04-22 16:23:01.124157] WDIAG [STORAGE.TRANS] on_timeout (ob_gts_rpc.h:168) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] gts rpc timeout(dst="172.23.88.130:2882", tenant_id=1) [2024-04-22 16:23:01.124183] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774180780206]}) [2024-04-22 16:23:01.152485] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:672) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=20] failed to get resp and destroy client, addr: "172.23.88.130:2882", last_write_ts_=1713774178151242, last_read_ts_=1713774177951202 [2024-04-22 16:23:01.157833] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10606][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=59][errcode=0] server is initiating(server_id=0, local_seq=14745, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:01.157894] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10606][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=56] new mysql sessid created(conn.sessid_=3221240217, support_ssl=false) [2024-04-22 16:23:01.157968] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10606][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221240217, sess.client_addr_="172.23.88.134:41856") [2024-04-22 16:23:01.158017] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10606][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=35] accept one succ(*s={this:0x7f3cf8b72970, session_id:3221240217, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:194}, err:0, last_decode_time:0, last_write_time:1713774181157960, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:01.158366] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10606][sql_nio5][T1][Y0-0000000000000000-0-0] [lt=113] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:01.158402] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [10606][sql_nio5][T1][Y0-0000000000000000-0-0] [lt=33] mysql login queue(mysql_queue->queue_.size()=0) [2024-04-22 16:23:01.158420] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=4] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:01.158941] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=22] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:01.158966] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=20] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:01.158978] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=11][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"H4h2\Y*e>G)4k?75E=?c"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:23:01.158995] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=15][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"H4h2\Y*e>G)4k?75E=?c"}, ret=-4043) [2024-04-22 16:23:01.159008] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=7][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:23:01.159018] WDIAG [SERVER] process (obmp_connect.cpp:346) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=6][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:23:01.159035] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=4] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:01.159100] INFO [SERVER] free_session (obmp_base.cpp:326) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=20] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221240217, proxy_sessid:0}) [2024-04-22 16:23:01.159131] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=27][errcode=0] server close connection(sessid=3221240217, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:01.159140] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221240217, proxy_sessid=0) [2024-04-22 16:23:01.159145] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=4][errcode=-4016] session is null [2024-04-22 16:23:01.159151] INFO [SERVER] process (obmp_connect.cpp:488) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=4] MySQL LOGIN(direct_client_ip="172.23.88.134", client_ip=172.23.88.134, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221240217, proxy_sessid=0, client_sessid=3221240217, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=true, capability=683647754, proxy_capability=1480, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:23:01.159169] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221240217, proxy_sessid=0) [2024-04-22 16:23:01.159173] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=4][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:01.159194] WDIAG [SERVER] response (obmp_base.cpp:89) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=8][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:01.159208] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDA-0-0] [lt=13][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:01.159264] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10606][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f3cf8b72970, session_id:3221240217, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:194}, err:5, last_decode_time:1713774181158251, last_write_time:1713774181159261, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:377, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:01.159328] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10606][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=52] kill and revert session(conn.sessid_=3221240217, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:01.159349] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10606][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f3cf8b72970, session_id:3221240217, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:194}, err:5, last_decode_time:1713774181158251, last_write_time:1713774181159261, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:377, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:01.159427] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10606][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=40] connection close(sessid=3221240217, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:01.162520] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=42] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:01.163792] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=26] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:01.163841] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED34-0-0] [lt=27] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:01.164808] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=3483631, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.164839] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.194328] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=79][errcode=0] server is initiating(server_id=0, local_seq=14746, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:01.194501] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=167] new mysql sessid created(conn.sessid_=3221240218, support_ssl=false) [2024-04-22 16:23:01.194585] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221240218, sess.client_addr_="172.23.88.134:41858") [2024-04-22 16:23:01.194648] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=47] accept one succ(*s={this:0x7f3ce8bfc030, session_id:3221240218, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:201}, err:0, last_decode_time:0, last_write_time:1713774181194578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:01.195068] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10607][sql_nio6][T1][Y0-0000000000000000-0-0] [lt=68] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:01.195116] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=11] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:01.195602] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=10] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:01.195626] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=20] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:01.195635] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=8][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"Wi.FUX;_J7qW=j!bL7j5"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:23:01.195647] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=11][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"Wi.FUX;_J7qW=j!bL7j5"}, ret=-4043) [2024-04-22 16:23:01.195655] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=7][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:23:01.195666] WDIAG [SERVER] process (obmp_connect.cpp:346) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=10][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:23:01.195682] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=4] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:01.195728] INFO [SERVER] free_session (obmp_base.cpp:326) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=9] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221240218, proxy_sessid:0}) [2024-04-22 16:23:01.195747] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=15][errcode=0] server close connection(sessid=3221240218, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:01.195754] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=6][errcode=-4018] get session fail(ret=-4018, sessid=3221240218, proxy_sessid=0) [2024-04-22 16:23:01.195782] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=27][errcode=-4016] session is null [2024-04-22 16:23:01.195790] INFO [SERVER] process (obmp_connect.cpp:488) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.23.88.134", client_ip=172.23.88.134, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221240218, proxy_sessid=0, client_sessid=3221240218, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=true, capability=683647754, proxy_capability=1480, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:23:01.195806] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221240218, proxy_sessid=0) [2024-04-22 16:23:01.195814] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=5][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:01.195830] WDIAG [SERVER] response (obmp_base.cpp:89) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=4][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:01.195847] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDC-0-0] [lt=16][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:01.195922] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4015] close sql sock by user req(*s={this:0x7f3ce8bfc030, session_id:3221240218, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:201}, err:5, last_decode_time:1713774181194999, last_write_time:1713774181195919, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:01.196026] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=90] kill and revert session(conn.sessid_=3221240218, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:01.196075] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=46] can close safely, do destroy(*s={this:0x7f3ce8bfc030, session_id:3221240218, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:201}, err:5, last_decode_time:1713774181194999, last_write_time:1713774181195919, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:01.196174] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=95] connection close(sessid=3221240218, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:01.197796] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20719665152, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:01.210186] INFO keepalive_check (time_wheel.c:71) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO socket dest server in blacklist, it will be destroyed, sock=(ptr=0x7f3d01be69b8,dest=-2108155988:2882) [2024-04-22 16:23:01.210222] INFO eloop_handle_sock_event (eloop.c:95) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=21] PNIO sock destroy: sock=0x7f3d01be69b8, connection=fd:208:local:172.23.88.129:52392:remote:172.23.88.130:2882, s->mask=0xc [2024-04-22 16:23:01.210246] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO sk_destroy: s=0x7f3d01be69b8 io=0x7f3cfff84608 [2024-04-22 16:23:01.210252] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO resp_cb on sk_destroy: packet_id=22763 s=0x7f3d01be69b8 [2024-04-22 16:23:01.210261] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:40) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4121] pnio error(buf=NULL, sz=0, io_err=-46) [2024-04-22 16:23:01.210338] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6][errcode=-4121] sync rpc execute fail(ret=-4121, addr="172.23.88.130:2882", pcode=1308, timeout=1999428) [2024-04-22 16:23:01.210381] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=22][errcode=-4121] rpc task_execute fail(ret=-4121, tenant_id=1, svr="172.23.88.130:2882", timeout=1999428, timeout_timestamp=1713774181305492) [2024-04-22 16:23:01.210403] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:250) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=20][errcode=-4121] task execute failed(ret=-4121) [2024-04-22 16:23:01.210424] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=16][errcode=-4121] execute with sql failed(ret=-4121) [2024-04-22 16:23:01.210430] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4121] fail execute plan(ret=-4121) [2024-04-22 16:23:01.210464] WDIAG [SQL] open (ob_result_set.cpp:158) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=31][errcode=-4121] execute plan failed(ret=-4121) [2024-04-22 16:23:01.210503] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4121) [2024-04-22 16:23:01.210523] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=17][errcode=-4121] open result set failed(ret=-4121) [2024-04-22 16:23:01.210532] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=7][errcode=-4121] result set open failed(ret=-4121, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:01.210544] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=11][errcode=-4121] execute failed(ret=-4121, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.210558] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=12][errcode=-4121] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4121, err_:"OB_RPC_SEND_ERROR", retry_type:0, client_ret:-4121}, need_retry=false) [2024-04-22 16:23:01.210579] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=12][errcode=-4121] result set close failed(ret=-4121) [2024-04-22 16:23:01.210584] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=4][errcode=-4121] result set close failed(ret=-4121) [2024-04-22 16:23:01.210587] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=3][errcode=-4121] failed to close result(close_ret=-4121, ret=-4121) [2024-04-22 16:23:01.210600] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=4][errcode=-4121] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4121, ret=-4121) [2024-04-22 16:23:01.210608] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6] slow inner sql(last_ret=-4121, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1904612) [2024-04-22 16:23:01.210615] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=7][errcode=-4121] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4121, ret=-4121) [2024-04-22 16:23:01.210626] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4121] execute sql failed(ret=-4121, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:01.210632] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4121] retry_while_no_tenant_resource failed(ret=-4121, tenant_id=1) [2024-04-22 16:23:01.210639] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=7][errcode=-4121] execute_read failed(ret=-4121, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:01.210650] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=10][errcode=-4121] failed to read without check sys variable(ret=-4121, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:01.210660] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=7][errcode=-4121] execute sql failed(ret=-4121, ret="OB_RPC_SEND_ERROR", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:01.210715] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6][errcode=-4121] load failed(ret=-4121, for_update=false) [2024-04-22 16:23:01.210728] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=12][errcode=-4121] core_table load failed(ret=-4121, ret="OB_RPC_SEND_ERROR") [2024-04-22 16:23:01.210738] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=10][errcode=-4121] get failed(ret=-4121) [2024-04-22 16:23:01.210744] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4121] get_baseline_schema_version failed(ret=-4121, ret="OB_RPC_SEND_ERROR", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:01.210752] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=7][errcode=-4121] get baseline schema version failed(ret=-4121, ret="OB_RPC_SEND_ERROR", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:01.210758] WDIAG [SHARE.SCHEMA] operator() (ob_multi_version_schema_service.cpp:2425) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4121] fail to get baseline_schema_version(ret=-4121) [2024-04-22 16:23:01.210771] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2495) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=12] [REFRESH_SCHEMA] end refresh and add schema(ret=-4121, ret="OB_RPC_SEND_ERROR", tenant_ids=[], cost=1905287) [2024-04-22 16:23:01.210785] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:308) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=14][errcode=-4121] fail to refresh and add schema(ret=-4121, ret="OB_RPC_SEND_ERROR") [2024-04-22 16:23:01.210790] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:314) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5] [REFRESH_SCHEMA] end refresh schema with new mode, (ret=-4121, ret="OB_RPC_SEND_ERROR", tenant_ids=[], used time=1905319) [2024-04-22 16:23:01.211492] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:270) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=12][errcode=-4122] address in blacklist(ret=-4122, addr="172.23.88.130:2882") [2024-04-22 16:23:01.211512] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=19][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:01.211529] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=10][errcode=-4122] rpc task_execute fail(ret=-4122, tenant_id=1, svr="172.23.88.130:2882", timeout=1999389, timeout_timestamp=1713774183210808) [2024-04-22 16:23:01.211537] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:250) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=8][errcode=-4122] task execute failed(ret=-4122) [2024-04-22 16:23:01.211543] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4122] execute with sql failed(ret=-4122) [2024-04-22 16:23:01.211548] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=4][errcode=-4122] fail execute plan(ret=-4122) [2024-04-22 16:23:01.211558] WDIAG [SQL] open (ob_result_set.cpp:158) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=9][errcode=-4122] execute plan failed(ret=-4122) [2024-04-22 16:23:01.211581] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4122) [2024-04-22 16:23:01.211595] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=11][errcode=-4122] open result set failed(ret=-4122) [2024-04-22 16:23:01.211602] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6][errcode=-4122] result set open failed(ret=-4122, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:01.211616] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=13][errcode=-4122] execute failed(ret=-4122, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.211632] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=14][errcode=-4122] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4122, err_:"OB_RPC_POST_ERROR", retry_type:0, client_ret:-4122}, need_retry=false) [2024-04-22 16:23:01.211648] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=11][errcode=-4122] result set close failed(ret=-4122) [2024-04-22 16:23:01.211655] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=7][errcode=-4122] result set close failed(ret=-4122) [2024-04-22 16:23:01.211659] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=4][errcode=-4122] failed to close result(close_ret=-4122, ret=-4122) [2024-04-22 16:23:01.211674] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBED-0-0] [lt=4][errcode=-4122] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4122, ret=-4122) [2024-04-22 16:23:01.211681] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6][errcode=-4122] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4122, ret=-4122) [2024-04-22 16:23:01.211692] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=10][errcode=-4122] execute sql failed(ret=-4122, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:01.211697] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4122] retry_while_no_tenant_resource failed(ret=-4122, tenant_id=1) [2024-04-22 16:23:01.211702] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=4][errcode=-4122] execute_read failed(ret=-4122, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:01.211707] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=4][errcode=-4122] failed to read without check sys variable(ret=-4122, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:01.211720] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=11][errcode=-4122] execute sql failed(ret=-4122, ret="OB_RPC_POST_ERROR", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:01.211754] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6][errcode=-4122] load failed(ret=-4122, for_update=false) [2024-04-22 16:23:01.211759] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4122] core_table load failed(ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.211769] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=9][errcode=-4122] get failed(ret=-4122) [2024-04-22 16:23:01.211773] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=4][errcode=-4122] get_baseline_schema_version failed(ret=-4122, ret="OB_RPC_POST_ERROR", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:01.211779] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6][errcode=-4122] get baseline schema version failed(ret=-4122, ret="OB_RPC_POST_ERROR", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:01.211785] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:521) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=6][errcode=0] fail to update baseline schema version(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", *tenant_id=1) [2024-04-22 16:23:01.211791] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:236) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBED-0-0] [lt=5][errcode=-4121] fail to process refresh task(ret=-4121, ret="OB_RPC_SEND_ERROR", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-04-22 16:23:01.211802] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:599) [10477][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4121] fail to batch process task(ret=-4121) [2024-04-22 16:23:01.211807] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:499) [10477][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4121] fail to batch execute task(ret=-4121, tasks.count()=1) [2024-04-22 16:23:01.234883] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.234948] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=55][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}, cluster_id=1713772869, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.234980] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, cluster_id=1713772869, tenant_id=1, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.235002] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=20][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:01.235019] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1713772869, addr="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}) [2024-04-22 16:23:01.235797] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}) [2024-04-22 16:23:01.235843] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=34][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:01.237175] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.237254] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=76][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.237302] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=46] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.237345] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=39] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2024-04-22 16:23:01.238377] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=7][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:01.238422] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=30] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:01.238462] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=36][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:01.238472] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=8][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:01.238481] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:01.238501] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:01.238508] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:01.238518] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:01.238526] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:01.238532] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:01.238538] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:01.238544] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:01.238550] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:01.238557] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:01.238565] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:01.238582] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=7][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:01.238591] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.238601] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=7][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.238615] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:01.238623] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:01.238632] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:01.238639] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.238655] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=11][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:01.238670] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.238676] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.238682] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:01.238696] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:01.238702] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:01.238710] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:01.238723] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:01.238730] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:01.238737] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1498050, start=1713774181238179, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:01.238747] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=9][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:01.238754] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:01.238821] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=7][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:01.238831] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=8][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:01.238839] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=7][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:01.238845] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:01.238852] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=5][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:01.238861] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=7][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:01.238874] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED76-0-0] [lt=12][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:01.260559] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] Cache replace map node details(ret=0, replace_node_count=0, replace_time=5523, replace_start_pos=692054, replace_num=62914) [2024-04-22 16:23:01.260588] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:01.264614] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2024-04-22 16:23:01.264879] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:01.264943] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774181264867) [2024-04-22 16:23:01.264989] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774181064771, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:01.265024] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:01.265109] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] get wrs ts(ls_id={id:1}, delta=3584346, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.265156] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.280782] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.280826] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4122] post gts request failed(ret=-4122, ret="OB_RPC_POST_ERROR", server="172.23.88.130:2882", msg={tenant_id:1, srr:[mts=1713774181280760], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:01.280840] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4122] post gts request failed(ret=-4122, ret="OB_RPC_POST_ERROR", leader="172.23.88.130:2882", msg={tenant_id:1, srr:[mts=1713774181280760], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:01.280870] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774181280760]}) [2024-04-22 16:23:01.280880] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:175) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4122] refresh gts failed(ret=-4122, ret="OB_RPC_POST_ERROR", gts_tenant_info={v:1}) [2024-04-22 16:23:01.280891] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4122, ret="OB_RPC_POST_ERROR", gts_tenant_info={v:1}) [2024-04-22 16:23:01.280900] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFB78-0-0] [lt=20][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713774181280849}) [2024-04-22 16:23:01.297650] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:01.297680] WDIAG [SQL.ENG] query_statistics (ob_px_tenant_target_monitor.cpp:275) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4122] send rpc to query statistics failed, need send refresh all(ret=-4122) [2024-04-22 16:23:01.297688] WDIAG [SQL.ENG] refresh_statistics (ob_px_tenant_target_monitor.cpp:114) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4122] query statistics failed(ret=-4122) [2024-04-22 16:23:01.297694] WDIAG [COMMON] operator() (ob_px_target_mgr.h:158) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4122] target monitor refresh statistics failed(ret=-4122, px_tenant_info={v:1}, px_res_info->get_target_monitor()={is_init:true, tenant_id:1, server:"172.23.88.129:2882", dummy_cache_leader:"172.23.88.130:2882", role:2}) [2024-04-22 16:23:01.300379] INFO keepalive_check (time_wheel.c:71) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO socket dest server in blacklist, it will be destroyed, sock=(ptr=0x7f3cd3205358,dest=-2108155988:2882) [2024-04-22 16:23:01.300416] INFO eloop_handle_sock_event (eloop.c:95) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO sock destroy: sock=0x7f3cd3205358, connection=fd:204:local:172.23.88.129:52394:remote:172.23.88.130:2882, s->mask=0xc [2024-04-22 16:23:01.300475] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] PNIO sk_destroy: s=0x7f3cd3205358 io=0x7f3d01784608 [2024-04-22 16:23:01.322733] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.364983] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45] get wrs ts(ls_id={id:1}, delta=3684120, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.365017] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.398113] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20721762304, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:01.460732] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:01.465029] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:01.465079] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774181465022) [2024-04-22 16:23:01.465090] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774181265013, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:01.465125] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=3784091, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.465146] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.565157] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:01.565242] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=81][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774181565142) [2024-04-22 16:23:01.565274] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774181465099, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:01.565351] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] get wrs ts(ls_id={id:1}, delta=3884046, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.565399] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.598487] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20721762304, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:01.606680] INFO keepalive_check (time_wheel.c:71) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO socket dest server in blacklist, it will be destroyed, sock=(ptr=0x7f3cd3372048,dest=-2108155988:2882) [2024-04-22 16:23:01.606785] INFO eloop_handle_sock_event (eloop.c:95) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=69] PNIO sock destroy: sock=0x7f3cd3372048, connection=fd:189:local:172.23.88.129:52382:remote:172.23.88.130:2882, s->mask=0xc [2024-04-22 16:23:01.606870] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO sk_destroy: s=0x7f3cd3372048 io=0x7f3d00584608 [2024-04-22 16:23:01.606928] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=56] PNIO resp_cb on sk_destroy: packet_id=22769 s=0x7f3cd3372048 [2024-04-22 16:23:01.606944] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:40) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4121] pnio error(buf=NULL, sz=0, io_err=-46) [2024-04-22 16:23:01.606982] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO resp_cb on sk_destroy: packet_id=22748 s=0x7f3cd3372048 [2024-04-22 16:23:01.607010] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:40) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4121] pnio error(buf=NULL, sz=0, io_err=-46) [2024-04-22 16:23:01.607003] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4121] sync rpc execute fail(ret=-4121, addr="172.23.88.130:2882", pcode=5407, timeout=2000000) [2024-04-22 16:23:01.607036] WDIAG [CLOG] post_sync_request_to_server_ (log_net_service.h:347) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4121] ObLogRpc post_sync_request failed(ret=-4121, palf_id=1, req={get_type:1}, server="172.23.88.130:2882") [2024-04-22 16:23:01.607046] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5195) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4121] get_palf_max_scn failed(ret=-4121, palf_id=1) [2024-04-22 16:23:01.607071] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5155) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4121] get_palf_max_scn failed(ret=-4121, self="172.23.88.129:2882", palf_id=1) [2024-04-22 16:23:01.607071] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4121] sync rpc execute fail(ret=-4121, addr="172.23.88.130:2882", pcode=5392, timeout=10000000) [2024-04-22 16:23:01.607088] WDIAG [PALF] runTimerTask (log_updater.cpp:109) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] update_palf cost too much time(ret=0, cost_time_us=879663, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-04-22 16:23:01.607095] WDIAG [CLOG] handle_ls_array_ (ob_garbage_collector.cpp:168) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] query_is_valid_member failed(tmp_ret=-4121, leader="172.23.88.130:2882", request={tenant_id_:1, self_addr_:"172.23.88.129:2882", ls_array_:[{id:1}]}) [2024-04-22 16:23:01.607268] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1441) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=24] gc_check_member_list_ cost time(ret=0, time_us=3489986) [2024-04-22 16:23:01.607372] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1713) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=77] execute_gc cost time(ret=0, time_us=1) [2024-04-22 16:23:01.607493] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1636) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=41] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-22 16:23:01.607586] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1658) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=66] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-04-22 16:23:01.607648] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1713) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=57] execute_gc cost time(ret=0, time_us=65) [2024-04-22 16:23:01.607717] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=64] ObSafeDestroyHandler start process [2024-04-22 16:23:01.607744] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=23] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2024-04-22 16:23:01.607783] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=17] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2024-04-22 16:23:01.608182] INFO keepalive_check (time_wheel.c:71) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO socket dest server in blacklist, it will be destroyed, sock=(ptr=0x7f3d01ba4048,dest=-2108155988:2882) [2024-04-22 16:23:01.608230] INFO eloop_handle_sock_event (eloop.c:95) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] PNIO sock destroy: sock=0x7f3d01ba4048, connection=fd:198:local:172.23.88.129:52388:remote:172.23.88.130:2882, s->mask=0xc [2024-04-22 16:23:01.608275] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO sk_destroy: s=0x7f3d01ba4048 io=0x7f3d01184608 [2024-04-22 16:23:01.608282] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO resp_cb on sk_destroy: packet_id=22773 s=0x7f3d01ba4048 [2024-04-22 16:23:01.608299] WDIAG [STORAGE.TRANS] on_timeout (ob_gts_rpc.h:168) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] gts rpc timeout(dst="172.23.88.130:2882", tenant_id=1) [2024-04-22 16:23:01.608325] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774181280760]}) [2024-04-22 16:23:01.660909] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:01.665246] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=109] get wrs ts(ls_id={id:1}, delta=3984367, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.665283] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.707059] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.707060] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.707497] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.707772] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.707819] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.707858] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.707905] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.708246] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.708465] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.709004] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.710602] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.711976] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.712083] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=54] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.712235] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.712737] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2024-04-22 16:23:01.712762] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:01.712788] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=14] cannot_recycle_log_size statistics(cannot_recycle_log_size=356459, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:01.712976] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=47] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.713202] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.713618] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.713977] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.715015] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=23] gc stale ls task succ [2024-04-22 16:23:01.715452] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.716276] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=10026, mem_used=76950528) [2024-04-22 16:23:01.716788] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=75] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.717053] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:01.717489] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=40] current monitor number(seq_=-1) [2024-04-22 16:23:01.717628] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.717711] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.717977] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:01.726824] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:01.729023] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:01.729051] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:01.729065] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:01.729074] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:01.738081] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4012] sync rpc execute fail(ret=-4012, addr="172.23.88.130:2882", pcode=258, timeout=2000000) [2024-04-22 16:23:01.738118] WDIAG [SERVER] do_renew_lease (ob_lease_state_mgr.cpp:414) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4012] can't get lease from rs(rs_addr="172.23.88.130:2882", ret=-4012) [2024-04-22 16:23:01.738145] WDIAG [SERVER] renew_lease (ob_lease_state_mgr.cpp:261) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] do_renew_lease failed(ret=-4012) [2024-04-22 16:23:01.738206] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.738237] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}, cluster_id=1713772869, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.738249] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, cluster_id=1713772869, tenant_id=1, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.738263] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:01.738269] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1713772869, addr="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}) [2024-04-22 16:23:01.738916] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}) [2024-04-22 16:23:01.738936] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:01.738955] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.739017] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.739050] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=32] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.739091] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=39] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2024-04-22 16:23:01.739101] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:355) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.739139] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=37] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1713772869) [2024-04-22 16:23:01.739148] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:311) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2024-04-22 16:23:01.739155] WDIAG [SERVER] renew_lease (ob_lease_state_mgr.cpp:264) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4638] renew_master_rootserver failed(ret=-4638) [2024-04-22 16:23:01.739240] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.739255] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}, cluster_id=1713772869, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.739268] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, cluster_id=1713772869, tenant_id=1, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.739275] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:01.739280] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1713772869, addr="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}) [2024-04-22 16:23:01.739826] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.23.88.130:2882", cluster_id:1713772869}) [2024-04-22 16:23:01.739856] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:01.739868] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.739903] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:841) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", find_index=0, role=2, proposal_id_=6, *this={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.739931] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:6, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:01.739973] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=38] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2024-04-22 16:23:01.740007] INFO [SHARE.LOCATION] merge_leader_from (ob_location_struct.cpp:488) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] [LS_LOCATION] leader will be changed(new_leader={server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}, old_leader=NULL, exist_replica=NULL, old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774181740004, replica_locations:[{server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}, {server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}]}) [2024-04-22 16:23:01.740047] TRACE [TRACE] renew_lease (ob_lease_state_mgr.cpp:287) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=30] [slow heartbeat](TRACE=begin_ts=1713774179742633 2024-04-22 08:22:59.742633|[renew_lease_begin] u=0 |[do_renew_lease_begin] u=3 |[send_heartbeat_begin] u=91 |[send_heartbeat_end] u=1995390 ret:-4012|[do_renew_lease_end] u=12 ret:-4012|[renew_master_rs_begin] u=22 |[renew_lease_end] u=1895 ret:-4638|total_timeu=1997413) [2024-04-22 16:23:01.751848] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6A-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.751880] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6A-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.129:2882"}, cluster_id=1713772869, tenant_id=1, group_id=10, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.751909] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6A-0-0] [lt=28][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, cluster_id=1713772869, tenant_id=1, arg={addr:"172.23.88.129:2882"}, group_id=10, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:01.751934] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6A-0-0] [lt=23][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:01.751962] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:857) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6A-0-0] [lt=24][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", addr="172.23.88.130:2882", timeout=2000000) [2024-04-22 16:23:01.752643] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6A-0-0] [lt=11][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=1193, arg={addr:"172.23.88.129:2882"}) [2024-04-22 16:23:01.752701] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:875) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6A-0-0] [lt=43][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:01.760537] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=68] task handle constructed(*this={this:0x7f3ce8455c90, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:01.760714] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=35] task handle constructed(*this={this:0x7f3ce8455c90, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:01.761504] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=40][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:01.761537] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=22] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:01.761551] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:01.761558] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:01.761567] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:01.761574] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=4][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:01.761579] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:01.761587] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:01.761604] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=16][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:01.761609] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=4][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:01.761614] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=4][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:01.761622] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=7][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:01.761627] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:01.761641] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:01.761647] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.761655] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=7][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.761663] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=7][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:01.761673] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:01.761679] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=5][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:01.761687] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.761706] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:01.761719] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.761723] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.761762] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=38][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:01.761773] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC67-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:01.761783] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC67-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:01.761789] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:01.761816] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:01.761830] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:01.761838] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:01.761844] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd148e050, start=1713774181760657, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:01.761886] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181760629), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:01.762576] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=10][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:01.762613] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=31] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:01.762623] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:01.762630] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=7][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:01.762642] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:01.762648] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:01.762654] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:01.762662] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:01.762668] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:01.762672] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=3][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:01.762676] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=4][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:01.762681] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=4][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:01.762685] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:01.762698] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:01.762704] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.762714] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.762724] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:01.762729] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:01.762733] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=4][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:01.762738] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.762749] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=9][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:01.762759] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.762763] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.762773] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:01.762788] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC68-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:01.762796] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC68-0-0] [lt=6][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:01.762803] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:01.762820] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:01.762826] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:01.762831] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:01.762841] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd2268050, start=1713774181761942, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:01.762873] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181761931), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:01.763353] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=66][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.763368] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=12][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={BASE:{msg_type:"Accept Request", id:1, sender:"172.23.88.129:2882", receiver:"172.23.88.130:2882", restart_counter:1, ballot_number:10, debug_ts:{src_construct_ts:"23:01.763313", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1713774181763313}, biggest_min_cluster_version_ever_seen:4.3.0.1}, lease_start_ts_on_proposer:1432583880, lease_interval:4000000, membership_version:{proposal_id:6, config_seq:7}}, server="172.23.88.130:2882") [2024-04-22 16:23:01.763516] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=68][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:01.763533] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=16][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:01.763553] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=19][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:01.763565] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=11][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:01.764539] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=164] task handle constructed(*this={this:0x7f3ccba54b20, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:01.764607] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.764621] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=12][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={BASE:{msg_type:"Accept Request", id:1, sender:"172.23.88.129:2882", receiver:"172.23.88.130:2882", restart_counter:1, ballot_number:10, debug_ts:{src_construct_ts:"23:01.764569", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1713774181764569}, biggest_min_cluster_version_ever_seen:4.3.0.1}, lease_start_ts_on_proposer:1432585136, lease_interval:4000000, membership_version:{proposal_id:6, config_seq:7}}, server="172.23.88.130:2882") [2024-04-22 16:23:01.764733] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=68][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:01.764743] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:01.764756] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=12][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:01.764768] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=11][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:01.764905] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=89][errcode=0] compare between invalid priority [2024-04-22 16:23:01.764932] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10918][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=43][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:01.764948] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10918][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=14][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:01.764966] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10918][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=17][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:01.764979] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10918][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=12][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:01.765257] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=11][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:01.765260] WDIAG [PALF] need_update_leader_ (log_state_mgr.cpp:958) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=52][errcode=0] leader or epoch has changed, need update(bool_ret=true, new_leader="172.23.88.129:2882", palf_id=1, new_leader_epoch=10, leader_="172.23.88.130:2882", leader_epoch_=9, self="172.23.88.129:2882") [2024-04-22 16:23:01.765288] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=25] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:01.765308] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=5][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:01.765296] INFO [PALF] reset_match_lsn_map_ (log_sliding_window.cpp:3987) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20] reset_match_lsn_map_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", member_list=3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, max_flushed_end_lsn={lsn:356581}, committed_end_lsn={lsn:356459}) [2024-04-22 16:23:01.765301] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:01.765325] INFO [PALF] to_leader_reconfirm (log_sliding_window.cpp:2588) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=29] to_leader_reconfirm success(ret=0, palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:01.765326] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=24][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:01.765324] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=15][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:01.765333] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:01.765337] INFO [PALF] reset_status_ (log_state_mgr.cpp:721) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] reset_status_(palf_id=1, self="172.23.88.129:2882", leader_epoch=-1, leader_="0.0.0.0:0") [2024-04-22 16:23:01.765341] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:01.765335] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=10][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:01.765349] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:01.765355] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:01.765349] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=14][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:01.765360] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:01.765364] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:01.765369] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:01.765373] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:01.765357] INFO [PALF] clean_log_ (log_sliding_window.cpp:2536) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] log_task is valid, check if need clean(ret=0, tmp_log_id=2925, palf_id=1, self="172.23.88.129:2882", log_task={header:{begin_lsn:{lsn:356459}, end_lsn:{lsn:356581}, log_id:2925, min_scn:{val:1713774177880657607, v:0}, max_scn:{val:1713774177880657607, v:0}, data_len:66, proposal_id:6, prev_lsn:{lsn:356337}, prev_proposal_id:6, committed_end_lsn:{lsn:356459}, data_checksum:667309872, accum_checksum:1991013568, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774177887661, freeze_ts:1713774177887661, submit_ts:1713774177887674, flushed_ts:1713774177893377, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:5703}) [2024-04-22 16:23:01.765377] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:01.765380] INFO [PALF] clean_log_ (log_sliding_window.cpp:2544) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23] find last submit log_task(ret=0, tmp_log_id=2925, palf_id=1, self="172.23.88.129:2882", log_task={header:{begin_lsn:{lsn:356459}, end_lsn:{lsn:356581}, log_id:2925, min_scn:{val:1713774177880657607, v:0}, max_scn:{val:1713774177880657607, v:0}, data_len:66, proposal_id:6, prev_lsn:{lsn:356337}, prev_proposal_id:6, committed_end_lsn:{lsn:356459}, data_checksum:667309872, accum_checksum:1991013568, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774177887661, freeze_ts:1713774177887661, submit_ts:1713774177887674, flushed_ts:1713774177893377, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:5703}, last_submit_log_id=2925) [2024-04-22 16:23:01.765392] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:01.765341] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:01.765398] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.765404] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.765404] INFO [PALF] truncate (lsn_allocator.cpp:93) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18] truncate success(lsn={lsn:356581}, log_id=2925, scn={val:1713774177880657607, v:0}) [2024-04-22 16:23:01.765408] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:01.765411] INFO [PALF] truncate_lsn_allocator_ (log_sliding_window.cpp:4366) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6] lsn_allocator_.truncate success(ret=0, palf_id=1, self="172.23.88.129:2882", last_lsn={lsn:356581}, last_log_id=2925, last_scn={val:1713774177880657607, v:0}) [2024-04-22 16:23:01.765413] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:01.765401] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774181765327) [2024-04-22 16:23:01.765419] INFO [PALF] clean_log_ (log_sliding_window.cpp:2576) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6] clean log finished(ret=0, palf_id=1, self="172.23.88.129:2882", max_log_id=2925, first_empty_log_id=-1, last_submit_log_id=2925, start_log_id=2925, max_log_id=2925, new_last_log_id=2925, new_last_scn={val:1713774177880657607, v:0}, new_last_log_end_lsn={lsn:356581}) [2024-04-22 16:23:01.765426] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:01.765452] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=26][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.765460] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323B-0-0] [lt=84][errcode=0] compare between invalid priority [2024-04-22 16:23:01.765462] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=9][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:01.765477] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.765476] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] task handle constructed(*this={this:0x7f3ce7dcf510, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:01.765481] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.765429] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774181565310, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:01.765485] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:01.765495] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC69-0-0] [lt=5][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:01.765514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC69-0-0] [lt=18][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:01.765518] INFO [SHARE] add_event_with_retry (ob_event_history_table_operator.h:374) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] add_event_with_retry success(ret=0, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)) [2024-04-22 16:23:01.765526] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:01.765529] INFO [PALF] follower_active_to_reconfirm_ (log_state_mgr.cpp:620) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] [PALF_EVENT] follower_active_to_reconfirm(palf_id=1, ret=0, self="172.23.88.129:2882", new_leader_epoch=10, leader_="172.23.88.129:2882", pending_end_lsn={lsn:338140}) [2024-04-22 16:23:01.765539] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:01.765537] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] switch_state(ret=0, palf_id=1, old_role=2, role=1, old_state=2, state=3, old_leader="172.23.88.130:2882", leader="172.23.88.129:2882", old_leader_epoch=9, leader_epoch=10, need_next_loop=true, diff=250) [2024-04-22 16:23:01.765545] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:01.765538] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] get wrs ts(ls_id={id:1}, delta=4084539, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.765557] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:01.765566] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cbe430050, start=1713774181764600, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:01.765566] INFO [PALF] init_reconfirm_ (log_reconfirm.cpp:180) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11] init_reconfirm_ success(ret=0, palf_id=1, member_list=3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, self="172.23.88.129:2882", majority_cnt=2, replica_num=3) [2024-04-22 16:23:01.765583] INFO [PALF] reconfirm (log_reconfirm.cpp:477) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] [PALF_EVENT] Reconfirm come into WAITING_LOG_FLUSHED state(palf_id=1, self="172.23.88.129:2882", majority_cnt=2, curr_paxos_follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}) [2024-04-22 16:23:01.765569] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.765648] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774181764586), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:01.765647] INFO [PALF] submit_io_task (log_io_worker.cpp:149) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] submit flush meta task success(io_task={LogIOTask:{BaseClass:"LogIOTask", palf_id:1, palf_epoch:1, create_task_ts:1713774181765646, push_cb_into_cb_pool_ts:-1, submit_seq_:25}, purge_ctx:{purge_type:"PURGE_BY_RECONFIRM"}, is_inited:true}) [2024-04-22 16:23:01.765694] INFO [PALF] submit_purge_throttling_task (log_engine.cpp:587) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] submit_purge_throttling success(last_purge_throttling_ts_=1713774181764894, purge_type="PURGE_BY_RECONFIRM") [2024-04-22 16:23:01.765701] INFO [PALF] purge_throttling_ (log_reconfirm.cpp:652) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] submit_purge_throttling_task during reconfirming(palf_id=1) [2024-04-22 16:23:01.765707] INFO [PALF] is_all_log_flushed_ (log_sliding_window.cpp:2343) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5] is_all_log_flushed_(bool_ret=true, palf_id=1, self="172.23.88.129:2882", curr_end_lsn={lsn:356581}, max_flushed_end_lsn={lsn:356581}) [2024-04-22 16:23:01.765719] INFO [PALF] wait_all_log_flushed_ (log_reconfirm.cpp:253) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_log_flushed_ success(ret=0, palf_id=1) [2024-04-22 16:23:01.765734] INFO [PALF] submit_prepare_req_ (log_mode_mgr.cpp:613) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=4] submit_prepare_meta_req success(ret=0, palf_id=1, self="172.23.88.129:2882", follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, ack_list=1"172.23.88.129:2882";, new_proposal_id=7) [2024-04-22 16:23:01.765729] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=81, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:01.765759] INFO [PALF] submit_prepare_req_ (log_mode_mgr.cpp:613) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24] submit_prepare_meta_req success(ret=0, palf_id=1, self="172.23.88.129:2882", follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, ack_list=1"172.23.88.129:2882";, new_proposal_id=7) [2024-04-22 16:23:01.765777] INFO [PALF] handle_io_task_ (log_io_worker.cpp:227) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO STAT](do_task_used_ts=9199, do_task_count=3, average_cost_ts=3066, io_queue_size=0) [2024-04-22 16:23:01.765778] INFO [PALF] switch_state_ (log_mode_mgr.cpp:527) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15] change_access_mode waiting retry(ret=-4023, palf_id=1, state="MODE_PREPARE", follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt=2, ack_list=1"172.23.88.129:2882";) [2024-04-22 16:23:01.765819] INFO [PALF] serialize (log_meta.cpp:185) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] LogMeta serialize(*this={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:6, prev:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:5, prev_lsn:{lsn:338018}, prev_mode_pid:6}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, buf="", buf=0x7f3ce7dce800, pos=779) [2024-04-22 16:23:01.765873] INFO [PALF] serialize_log_meta_ (log_engine.cpp:1528) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=51] serialize_log_meta_ success(ret=0, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:6, prev:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:5, prev_lsn:{lsn:338018}, prev_mode_pid:6}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_entry={header:{magic:19533, version:1, data_len:4072, data_checksum:716023874, header_checksum:3693370758}, buf_:0x7f3ce7dce800}, log_meta_entry_header_len=24) [2024-04-22 16:23:01.765915] INFO [PALF] submit_io_task (log_io_worker.cpp:149) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=41] submit flush meta task success(io_task={LogIOTask:{BaseClass:"LogIOTask", palf_id:1, palf_epoch:1, create_task_ts:1713774181765791, push_cb_into_cb_pool_ts:-1, submit_seq_:26}, flush_meta_cb_ctx:{type:"PREPARE_META", proposal_id:7, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}}) [2024-04-22 16:23:01.765937] INFO [PALF] submit_flush_meta_task_ (log_engine.cpp:1287) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13] submit_flush_meta_task_ success(flush_meta_cb_ctx={type:"PREPARE_META", proposal_id:7, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:6, prev:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:5, prev_lsn:{lsn:338018}, prev_mode_pid:6}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}) [2024-04-22 16:23:01.765978] INFO [PALF] submit_flush_prepare_meta_task (log_engine.cpp:395) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=40] submit_flush_prepare_meta_task success(ret=0, flush_meta_cb_ctx={type:"PREPARE_META", proposal_id:7, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}, prepare_meta={version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}) [2024-04-22 16:23:01.765996] INFO [PALF] write_prepare_meta_ (log_state_mgr.cpp:548) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17] write_prepare_meta_ success(prepare_meta={version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, new_leader="172.23.88.129:2882", prepare_meta_={version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}) [2024-04-22 16:23:01.766005] INFO [PALF] handle_prepare_request (log_state_mgr.cpp:468) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] LogStateMgr handle_prepare_request success(ret=0, palf_id=1, self_="172.23.88.129:2882", server="172.23.88.129:2882", proposal_id=7, leader_="172.23.88.129:2882", broadcast_leader="0.0.0.0:0") [2024-04-22 16:23:01.766040] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.766050] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={log_proposal_id:7}, server="172.23.88.130:2882") [2024-04-22 16:23:01.766083] INFO [PALF] submit_prepare_meta_req (log_net_service.h:135) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] submit_prepare_meta_req success(ret=0, member_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}) [2024-04-22 16:23:01.766092] INFO [PALF] submit_prepare_log_ (log_reconfirm.cpp:240) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] submit_prepare_meta_req success(ret=0, palf_id=1, curr_paxos_follower_list_=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, prepare_log_ack_list_=0, old_proposal_id=6, new_proposal_id_=7, self_="172.23.88.129:2882", majority_max_lsn_={lsn:356581}, majority_max_log_server_="172.23.88.129:2882", majority_max_accept_pid_=6, now_us=1713774181765723, old_prepare_time_us=-1, last_submit_prepare_req_time_us_=1713774181765723, max_flushed_log_pid=6, local_accept_proposal_id=6) [2024-04-22 16:23:01.766109] INFO [PALF] reconfirm (log_reconfirm.cpp:494) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17] [PALF_EVENT] Reconfirm come into FETCH_MAX_LOG_LSN state(palf_id=1, self="172.23.88.129:2882", majority_max_accept_pid=6) [2024-04-22 16:23:01.766120] INFO [PALF] reconfirm (log_reconfirm.cpp:638) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] reconfirm waiting retry(ret=-4023, palf_id=1, state=2) [2024-04-22 16:23:01.766125] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=4] switch_state(ret=-4023, palf_id=1, old_role=1, role=1, old_state=3, state=3, old_leader="172.23.88.129:2882", leader="172.23.88.129:2882", old_leader_epoch=10, leader_epoch=10, need_next_loop=false, diff=576) [2024-04-22 16:23:01.766867] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:01.766925] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=42] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:01.766946] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=18][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:01.766957] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:01.766968] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=9][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:01.766980] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:01.766987] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:01.766997] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:01.767058] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=59][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:01.767075] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=15][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:01.767086] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=11][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:01.767095] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=8][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:01.767105] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:01.767129] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:01.767141] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.767155] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.767167] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:01.767176] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:01.767186] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=8][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:01.767193] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.767237] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=39][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:01.767254] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.767261] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.767266] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:01.767282] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10480][Occam][T1][YB42AC175881-000616AB083FEC5F-0-0] [lt=6][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:01.767292] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10480][Occam][T0][YB42AC175881-000616AB083FEC5F-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:01.767301] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)) [2024-04-22 16:23:01.767352] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:01.767360] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:01.767372] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)") [2024-04-22 16:23:01.767385] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd227a050, start=1713774181765580, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)) [2024-04-22 16:23:01.767495] WDIAG [SHARE] operator() (ob_event_history_table_operator.cpp:393) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] sync execute sql failed(sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)", ret=-5019) [2024-04-22 16:23:01.769128] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=94208, this={dio_aligned_buf:{buf_write_offset:0, buf_padding_size:0, align_size:4096, aligned_buf_size:8192, aligned_used_ts:235, truncate_used_ts:9}, log_block_size:67108864, dir_fd:234, io_fd:236}, aligned_buf_len=4096, aligned_buf="LM", aligned_block_offset=94208, buf_len=4096, total_write_size_=12288, total_write_size_after_dio_=12288, ob_pwrite_used_ts=12011, count=3, each_pwrite_cost=4003) [2024-04-22 16:23:01.769164] INFO [PALF] append_meta (log_storage.cpp:269) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=33] LogStorage append meta success(ret=0, log_block_header_={magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, log_tail_={lsn:94208}, this={log_tail:{lsn:94208}, readable_log_tail:{lsn:94208}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67010560, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}) [2024-04-22 16:23:01.769210] INFO [PALF] inner_after_flush_meta (palf_handle_impl.cpp:4174) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=58] inner_after_flush_meta(flush_meta_cb_ctx={type:"PREPARE_META", proposal_id:7, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}) [2024-04-22 16:23:01.769229] INFO [PALF] after_flush_prepare_meta_ (palf_handle_impl.cpp:4232) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=17] do not need submit prepare response to self(self="172.23.88.129:2882", leader:="172.23.88.129:2882") [2024-04-22 16:23:01.771902] INFO [PALF] handle_prepare_response (log_mode_mgr.cpp:675) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D90-0-0] [lt=100] handle_prepare_response finish(ret=0, palf_id=1, self="172.23.88.129:2882", server="172.23.88.131:2882", msg_proposal_id=7, accept_log_proposal_id=6, last_lsn={lsn:356581}, mode_meta={version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, ack_list=2"172.23.88.129:2882";"172.23.88.131:2882";, follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt=2, max_majority_accepted_mode_meta={version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, max_majority_accepted_pid=6, max_majority_lsn={lsn:356581}) [2024-04-22 16:23:01.771959] INFO [PALF] add_server (log_simple_member_list.cpp:158) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D90-0-0] [lt=52] add_server success(count=0, server="172.23.88.131:2882", count=0) [2024-04-22 16:23:01.771969] INFO [PALF] handle_prepare_response (log_reconfirm.cpp:323) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D90-0-0] [lt=9] handle_prepare_response finished(ret=0, palf_id=1, server="172.23.88.131:2882", accept_proposal_id=6, last_lsn={lsn:356581}, majority_max_accept_pid_=6, majority_max_lsn_={lsn:356581}, prepare_log_ack_list_=1"172.23.88.131:2882";) [2024-04-22 16:23:01.781390] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.781429] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4122] post gts request failed(ret=-4122, ret="OB_RPC_POST_ERROR", server="172.23.88.130:2882", msg={tenant_id:1, srr:[mts=1713774181781375], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:01.781465] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4122] post gts request failed(ret=-4122, ret="OB_RPC_POST_ERROR", leader="172.23.88.130:2882", msg={tenant_id:1, srr:[mts=1713774181781375], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:01.781499] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774181781375]}) [2024-04-22 16:23:01.790875] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:01.790938] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=45] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:01.790971] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:01.790991] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:01.791014] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=21][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:01.791035] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=15][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:01.791051] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:01.791071] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:01.791083] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:01.791181] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=96][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:01.791199] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=16][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:01.791213] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=12][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:01.791225] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:01.791235] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:01.791245] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:01.791267] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:01.791277] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.791289] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:01.791299] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:01.791308] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:01.791320] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:01.791331] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.791387] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=50][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:01.791412] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.791421] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:01.791429] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:01.791506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7B-0-0] [lt=40][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:01.791522] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7B-0-0] [lt=13][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:01.791534] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7B-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:01.791546] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7B-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:01.791555] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7B-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:01.798301] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:01.798377] WDIAG [SQL.ENG] query_statistics (ob_px_tenant_target_monitor.cpp:275) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4122] send rpc to query statistics failed, need send refresh all(ret=-4122) [2024-04-22 16:23:01.798402] WDIAG [SQL.ENG] refresh_statistics (ob_px_tenant_target_monitor.cpp:114) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4122] query statistics failed(ret=-4122) [2024-04-22 16:23:01.798424] WDIAG [COMMON] operator() (ob_px_target_mgr.h:158) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4122] target monitor refresh statistics failed(ret=-4122, px_tenant_info={v:1}, px_res_info->get_target_monitor()={is_init:true, tenant_id:1, server:"172.23.88.129:2882", dummy_cache_leader:"172.23.88.130:2882", role:2}) [2024-04-22 16:23:01.798808] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:01.824679] INFO keepalive_check (time_wheel.c:71) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO socket dest server in blacklist, it will be destroyed, sock=(ptr=0x7f3cd33d4048,dest=-2108155988:2882) [2024-04-22 16:23:01.824826] INFO eloop_handle_sock_event (eloop.c:95) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=109] PNIO sock destroy: sock=0x7f3cd33d4048, connection=fd:188:local:172.23.88.129:52380:remote:172.23.88.130:2882, s->mask=0xc [2024-04-22 16:23:01.824978] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO sk_destroy: s=0x7f3cd33d4048 io=0x7f3d00d84608 [2024-04-22 16:23:01.825022] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO resp_cb on sk_destroy: packet_id=22767 s=0x7f3cd33d4048 [2024-04-22 16:23:01.825069] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=21] PNIO resp_cb on sk_destroy: packet_id=22746 s=0x7f3cd33d4048 [2024-04-22 16:23:01.825087] WDIAG [RPC] handle_resp (ob_poc_rpc_proxy.cpp:40) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4121] pnio error(buf=NULL, sz=0, io_err=-46) [2024-04-22 16:23:01.825139] WDIAG [RPC] send (ob_poc_rpc_proxy.h:168) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=1][errcode=-4121] sync rpc execute fail(ret=-4121, addr="172.23.88.130:2882", pcode=1282, timeout=11972849) [2024-04-22 16:23:01.825174] WDIAG [SQL.EXE] task_execute (ob_executor_rpc_impl.cpp:81) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=25][errcode=-4121] rpc task_execute fail(ret=-4121, tenant_id=1, svr="172.23.88.130:2882", timeout=11972849, timeout_timestamp=1713774189953195) [2024-04-22 16:23:01.825192] WDIAG [SQL.EXE] execute (ob_remote_task_executor.cpp:99) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=14][errcode=-4121] fail post task(ret=-4121) [2024-04-22 16:23:01.825205] WDIAG [SQL.EXE] execute (ob_remote_job_executor.cpp:55) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=11][errcode=-4121] fail execute task(ret=-4121, *task_info={task_loc:{server:"172.23.88.130:2882", ob_task_id:{ob_job_id:{ob_execution_id:{server:"172.23.88.129:2882", execution_id:18446744073709551615, task_type:0, hash:4219408603045787357}, job_id:19075}, task_id:0, task_cnt:0}}, range_location:{part_locs:[cnt:0], server:"0.0.0.0:0"}, location_idx:0, location_idx_list:[cnt:1, 0], state:5, slice_count_pos:[cnt:0], background:false, retry_times:0, location_idx_list:[cnt:1, 0]}) [2024-04-22 16:23:01.825228] WDIAG [SQL.EXE] execute_with_plan (ob_remote_scheduler.cpp:124) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=22][errcode=-4121] fail execute remote job(ret=-4121) [2024-04-22 16:23:01.825260] WDIAG [SQL.EXE] execute_with_plan (ob_remote_scheduler.cpp:141) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=18][errcode=-4121] fail to schedule, print job's status(ret=-4121, print_ret=0, job_status="{job_count:2, {task_count:0, state statistics:{}}, {task_count:1, task_info:{loc:"172.23.88.130:2882"ctrl:"172.23.88.129:2882"eid:18446744073709551615jid:19075tid:0pull_sid:0state:5}state statistics:{failed:1}}}") [2024-04-22 16:23:01.825270] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:57) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=6][errcode=-4121] execute with plan failed(ret=-4121) [2024-04-22 16:23:01.825276] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=5][errcode=-4121] fail execute plan(ret=-4121) [2024-04-22 16:23:01.825282] WDIAG [SQL] open (ob_result_set.cpp:158) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=4][errcode=-4121] execute plan failed(ret=-4121) [2024-04-22 16:23:01.825309] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4121) [2024-04-22 16:23:01.825324] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=13][errcode=-4121] open result set failed(ret=-4121) [2024-04-22 16:23:01.825331] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=6][errcode=-4121] result set open failed(ret=-4121, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:01.825340] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=8][errcode=-4121] execute failed(ret=-4121, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=221, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:01.825352] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=10][errcode=-4121] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:221, local_retry_times:221, err_:-4121, err_:"OB_RPC_SEND_ERROR", retry_type:0, client_ret:-4121}, need_retry=false) [2024-04-22 16:23:01.825373] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=8][errcode=-4121] result set close failed(ret=-4121) [2024-04-22 16:23:01.825379] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=6][errcode=-4121] result set close failed(ret=-4121) [2024-04-22 16:23:01.825388] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=9][errcode=-4121] failed to close result(close_ret=-4121, ret=-4121) [2024-04-22 16:23:01.825404] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=5][errcode=-4121] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4121, ret=-4121) [2024-04-22 16:23:01.825409] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=5] slow inner sql(last_ret=-4121, sql={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, process_time=21872196) [2024-04-22 16:23:01.825416] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCC-0-0] [lt=7][errcode=-4121] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-4121, ret=-4121) [2024-04-22 16:23:01.825424] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4121] execute sql failed(ret=-4121, tenant_id=1, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2024-04-22 16:23:01.825446] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4121] retry_while_no_tenant_resource failed(ret=-4121, tenant_id=1) [2024-04-22 16:23:01.825457] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4121] execute_read failed(ret=-4121, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:01.825463] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4121] query failed(ret=-4121, conn=0x7f3d1177a050, start=1713774159953195, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2024-04-22 16:23:01.825470] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4121] read failed(ret=-4121) [2024-04-22 16:23:01.825479] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:526) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4121] execute sql failed(ret=-4121, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2024-04-22 16:23:01.825555] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:185) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4121] fail to select for update snapshot_gc_scn(ret=-4121, ret="OB_RPC_SEND_ERROR", tenant_id=1) [2024-04-22 16:23:01.825562] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:597) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4121] failed to load updated info(ret=-4121) [2024-04-22 16:23:01.825578] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:669) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4121] fail to try update info(tmp_ret=-4121, tmp_ret="OB_RPC_SEND_ERROR") [2024-04-22 16:23:01.825593] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:704) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=5] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts=0) [2024-04-22 16:23:01.825600] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:731) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=6] update reserved snapshot finished(cost_ts=14, reserved_snapshot=0) [2024-04-22 16:23:01.827994] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFACA-0-0] [lt=92][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774153714643}) [2024-04-22 16:23:01.861050] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=58] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:01.865336] INFO [PALF] is_reconfirm_timeout_ (log_state_mgr.cpp:866) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15] leader reconfirm need wait(palf_id=1, start_id=2925, is_sw_timeout=false, self="172.23.88.129:2882", reconfirm=0x7f3cd2de84b0) [2024-04-22 16:23:01.865369] INFO [PALF] reconfirm (log_reconfirm.cpp:512) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24] [PALF_EVENT] Reconfirm come into RECONFIRM_MODE_META state(palf_id=1, prepare_log_ack_list_=1"172.23.88.131:2882";, majority_max_log_server="172.23.88.129:2882", majority_max_lsn={lsn:356581}) [2024-04-22 16:23:01.865448] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] get wrs ts(ls_id={id:1}, delta=4184416, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.865422] INFO [PALF] serialize (log_meta.cpp:185) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] LogMeta serialize(*this={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:6, prev:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:5, prev_lsn:{lsn:338018}, prev_mode_pid:6}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, buf="", buf=0x7f3ce7dce4d0, pos=779) [2024-04-22 16:23:01.865472] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.865491] INFO [PALF] serialize_log_meta_ (log_engine.cpp:1528) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=66] serialize_log_meta_ success(ret=0, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:6, prev:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:5, prev_lsn:{lsn:338018}, prev_mode_pid:6}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_entry={header:{magic:19533, version:1, data_len:4072, data_checksum:3360693219, header_checksum:406620905}, buf_:0x7f3ce7dce4d0}, log_meta_entry_header_len=24) [2024-04-22 16:23:01.865551] INFO [PALF] submit_io_task (log_io_worker.cpp:149) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=57] submit flush meta task success(io_task={LogIOTask:{BaseClass:"LogIOTask", palf_id:1, palf_epoch:1, create_task_ts:1713774181865409, push_cb_into_cb_pool_ts:-1, submit_seq_:27}, flush_meta_cb_ctx:{type:"MODE_META", proposal_id:7, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}}) [2024-04-22 16:23:01.865583] INFO [PALF] submit_flush_meta_task_ (log_engine.cpp:1287) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24] submit_flush_meta_task_ success(flush_meta_cb_ctx={type:"MODE_META", proposal_id:7, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:6, prev:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:5, prev_lsn:{lsn:338018}, prev_mode_pid:6}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}) [2024-04-22 16:23:01.865629] INFO [PALF] submit_flush_mode_meta_task (log_engine.cpp:437) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=45] submit_flush_mode_meta_task success(ret=0, flush_meta_cb_ctx={type:"MODE_META", proposal_id:7, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:01.865657] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.865670] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:false}, server="172.23.88.130:2882") [2024-04-22 16:23:01.865704] INFO [PALF] submit_accept_req_ (log_mode_mgr.cpp:633) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] submit_change_mode_meta_req success(ret=0, palf_id=1, self="172.23.88.129:2882", follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, ack_list=0, proposal_id=7, is_applied_mode_meta=false, mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:01.865740] INFO [PALF] switch_state_ (log_mode_mgr.cpp:527) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=34] change_access_mode waiting retry(ret=-4023, palf_id=1, state="MODE_ACCEPT", follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt=2, ack_list=0) [2024-04-22 16:23:01.865763] INFO [PALF] reconfirm (log_reconfirm.cpp:638) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22] reconfirm waiting retry(ret=-4023, palf_id=1, state=3) [2024-04-22 16:23:01.865774] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] switch_state(ret=-4023, palf_id=1, old_role=1, role=1, old_state=3, state=3, old_leader="172.23.88.129:2882", leader="172.23.88.129:2882", old_leader_epoch=10, leader_epoch=10, need_next_loop=false, diff=441) [2024-04-22 16:23:01.868452] INFO [PALF] append_meta (log_storage.cpp:269) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] LogStorage append meta success(ret=0, log_block_header_={magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, log_tail_={lsn:98304}, this={log_tail:{lsn:98304}, readable_log_tail:{lsn:98304}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67006464, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}) [2024-04-22 16:23:01.868515] INFO [PALF] inner_after_flush_meta (palf_handle_impl.cpp:4174) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=6] inner_after_flush_meta(flush_meta_cb_ctx={type:"MODE_META", proposal_id:7, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}) [2024-04-22 16:23:01.868551] INFO [PALF] after_flush_mode_meta (log_mode_mgr.cpp:771) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] after_flush_mode_meta success(ret=0, palf_id=1, self="172.23.88.129:2882", is_applied_mode_meta=false, mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, applied_mode_meta={version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:01.868569] INFO [PALF] ack_mode_meta (log_mode_mgr.cpp:796) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] ack_mode_meta finish(ret=0, palf_id=1, self="172.23.88.129:2882", server="172.23.88.129:2882", proposal_id=7, follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt=2, ack_list=1"172.23.88.129:2882";, resend_mode_meta_list={learner_num:0, learner_array:[]}) [2024-04-22 16:23:01.871067] INFO [PALF] ack_mode_meta (log_mode_mgr.cpp:796) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D91-0-0] [lt=13] ack_mode_meta finish(ret=0, palf_id=1, self="172.23.88.129:2882", server="172.23.88.131:2882", proposal_id=7, follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt=2, ack_list=2"172.23.88.129:2882";"172.23.88.131:2882";, resend_mode_meta_list={learner_num:0, learner_array:[]}) [2024-04-22 16:23:01.880549] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880612] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880634] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880650] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880664] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880758] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=91] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880776] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880789] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880818] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880840] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880865] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880899] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880916] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.880930] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.881011] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=78] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.881037] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:01.907756] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1713774181907754, bytes: 65549396, bw: 0.010306 MB/s, add_ts: 1000610, add_bytes: 10813 [2024-04-22 16:23:01.907903] INFO [PALF] handle (fetch_log_engine.cpp:284) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=16] handle fetch_log_task(fetch_log_task={timestamp_us:1713774181907864, id:1, server:"172.23.88.131:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:7, prev_lsn:{lsn:356337}, start_lsn:{lsn:356459}, log_size:41943040, log_count:2048, accepted_mode_pid:7}) [2024-04-22 16:23:01.907890] INFO [PALF] get_log (palf_handle_impl.cpp:3994) [11430][T1_L0_G1][T1][YB42AC175883-000616AB092732AE-0-0] [lt=28] submit fetch log task success(ret=0, palf_id=1, server="172.23.88.131:2882", fetch_type=0, prev_lsn={lsn:356337}, start_lsn={lsn:356459}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=7) [2024-04-22 16:23:01.907950] INFO [PALF] read (log_cache.cpp:100) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=488, hit_cnt=4, read_cnt=4, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:01.907978] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3768) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x7f3ccc7fd000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:356459}, end_lsn:{lsn:356581}, read_buf:{buf_len_:8192, buf_:0x7f3ccc7fd000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774177880657607, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:66, proposal_id:6, committed_lsn:{lsn:356459}, max_scn:{val:1713774177880657607, v:0}, accumulated_checksum:1991013568, log_id:2925, flag:0}}, init_mode_version:0, accumulate_checksum:1991013568, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774177880657607, v:0}}}, replayable_point={val:18446744073709551615, v:3}, prev_lsn={lsn:356459}) [2024-04-22 16:23:01.908068] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3705) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=42] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, prev_lsn={lsn:356337}, iterator={iterator_impl:{buf_:0x7f3ccc7fd000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:356459}, end_lsn:{lsn:356581}, read_buf:{buf_len_:8192, buf_:0x7f3ccc7fd000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774177880657607, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:66, proposal_id:6, committed_lsn:{lsn:356459}, max_scn:{val:1713774177880657607, v:0}, accumulated_checksum:1991013568, log_id:2925, flag:0}}, init_mode_version:0, accumulate_checksum:1991013568, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774177880657607, v:0}}}, fetch_log_count=2048, has_fetched_log_count=1) [2024-04-22 16:23:01.908125] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=28] submit_change_mode_meta_req success(ret=0, palf_id=1, server="172.23.88.131:2882", mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=7) [2024-04-22 16:23:01.908138] INFO [PALF] fetch_log_from_storage (palf_handle_impl.cpp:3490) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT FETCH LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=41943040) [2024-04-22 16:23:01.908161] INFO [LIB] stat (utility.h:1165) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FETCH LOG TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=190, this=0x7f3ce8ff4600, extra_info="") [2024-04-22 16:23:01.908169] INFO [LIB] stat (utility.h:1165) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FETCH LOG EXECUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=279, this=0x7f3ce8ff4830, extra_info="") [2024-04-22 16:23:01.908175] INFO [PALF] handle (fetch_log_engine.cpp:321) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=6] handle fetch log task(ret=0, palf_id=1, handle_cost_time_us=259, fetch_log_task={timestamp_us:1713774181907864, id:1, server:"172.23.88.131:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:7, prev_lsn:{lsn:356337}, start_lsn:{lsn:356459}, log_size:41943040, log_count:2048, accepted_mode_pid:7}, fetch_stat={total_size:122, group_log_cnt:1, read_cost:0, get_cost:0, send_cost:45}) [2024-04-22 16:23:01.909364] INFO [PALF] ack_mode_meta (log_mode_mgr.cpp:796) [11430][T1_L0_G1][T1][YB42AC175881-000616AB08EFEBCE-0-0] [lt=56] ack_mode_meta finish(ret=0, palf_id=1, self="172.23.88.129:2882", server="172.23.88.131:2882", proposal_id=7, follower_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt=2, ack_list=2"172.23.88.129:2882";"172.23.88.131:2882";, resend_mode_meta_list={learner_num:0, learner_array:[]}) [2024-04-22 16:23:01.921802] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1713774181921800, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2024-04-22 16:23:01.926527] INFO keepalive_check (time_wheel.c:71) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO socket dest server in blacklist, it will be destroyed, sock=(ptr=0x7f3cd33fe048,dest=-2108155988:2882) [2024-04-22 16:23:01.926609] INFO eloop_handle_sock_event (eloop.c:95) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO sock destroy: sock=0x7f3cd33fe048, connection=fd:195:local:172.23.88.129:52384:remote:172.23.88.130:2882, s->mask=0xc [2024-04-22 16:23:01.926804] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=120] PNIO sk_destroy: s=0x7f3cd33fe048 io=0x7f3cffb84608 [2024-04-22 16:23:01.926862] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO resp_cb on sk_destroy: packet_id=22771 s=0x7f3cd33fe048 [2024-04-22 16:23:01.965423] INFO [PALF] inc_update_scn_base (lsn_allocator.cpp:191) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13] inc_update_scn_base success(scn_base=1713774177880657607, scn_delta=0, scn=1) [2024-04-22 16:23:01.965492] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.965509] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:true}, server="172.23.88.130:2882") [2024-04-22 16:23:01.965506] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:01.965546] INFO [PALF] reconfirm (log_reconfirm.cpp:525) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] [PALF_EVENT] Reconfirm come into RECONFIRM_FETCH_LOG state(palf_id=1, prepare_log_ack_list_=1"172.23.88.131:2882";) [2024-04-22 16:23:01.965545] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774181965491) [2024-04-22 16:23:01.965566] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774181765489, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:01.965570] INFO [PALF] reconfirm (log_reconfirm.cpp:550) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19] [PALF_EVENT] Reconfirm come into RECONFIRMING state(palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:01.965578] INFO [PALF] reconfirm (log_reconfirm.cpp:638) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] reconfirm waiting retry(ret=-4023, palf_id=1, state=5) [2024-04-22 16:23:01.965586] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] switch_state(ret=-4023, palf_id=1, old_role=1, role=1, old_state=3, state=3, old_leader="172.23.88.129:2882", leader="172.23.88.129:2882", old_leader_epoch=10, leader_epoch=10, need_next_loop=false, diff=171) [2024-04-22 16:23:01.965603] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=4284387, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:01.965614] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:01.965620] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:true}, server="172.23.88.130:2882") [2024-04-22 16:23:01.965618] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:01.966848] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC11-0-0] [lt=27][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:01.966944] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC11-0-0] [lt=68] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:01.967012] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC11-0-0] [lt=61][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:01.999110] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:02.053262] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=30] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:02.061157] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:02.065492] INFO [PALF] is_reconfirm_timeout_ (log_state_mgr.cpp:866) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] leader reconfirm need wait(palf_id=1, start_id=2925, is_sw_timeout=false, self="172.23.88.129:2882", reconfirm=0x7f3cd2de84b0) [2024-04-22 16:23:02.065523] INFO [PALF] is_all_log_flushed_ (log_sliding_window.cpp:2343) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=28] is_all_log_flushed_(bool_ret=true, palf_id=1, self="172.23.88.129:2882", curr_end_lsn={lsn:356581}, max_flushed_end_lsn={lsn:356581}) [2024-04-22 16:23:02.065595] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=4385127, log_service_ts=1713774177680355, min_tx_service_ts=4611686018427387, timestamp={val:1713774177680355699, v:0}) [2024-04-22 16:23:02.065622] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:02.065695] INFO [PALF] update_election_meta_ (log_config_mgr.cpp:1834) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=95] update_election_meta_ success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", log_ms_meta={version:3, proposal_id:6, prev:{version:1, config:{config_version:{proposal_id:5, config_seq:6}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:5, prev_lsn:{lsn:338018}, prev_mode_pid:6}, new_election_member_list={addr_list:["172.23.88.129:2882", "172.23.88.130:2882", "172.23.88.131:2882"], membership_version:{proposal_id:7, config_seq:8}, replica_num:3}) [2024-04-22 16:23:02.065761] INFO [PALF] renew_config_change_barrier_ (log_config_mgr.cpp:1015) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=62] renew_config_change_barrier_ success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", checking_barrier={prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_end_lsn:{lsn:356581}, prev_mode_pid:7}) [2024-04-22 16:23:02.065805] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=488) [2024-04-22 16:23:02.065846] INFO [LIB] stat (utility.h:1165) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=25, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:02.065860] INFO [LIB] stat (utility.h:1165) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT LOG LIFE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=1097808, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:02.065870] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=4, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=29, avg_log_submit_to_flush_time=5150, avg_log_submit_to_slide_time=1097778) [2024-04-22 16:23:02.065879] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2925, log_task={header:{begin_lsn:{lsn:356459}, end_lsn:{lsn:356581}, log_id:2925, min_scn:{val:1713774177880657607, v:0}, max_scn:{val:1713774177880657607, v:0}, data_len:66, proposal_id:6, prev_lsn:{lsn:356337}, prev_proposal_id:6, committed_end_lsn:{lsn:356459}, data_checksum:667309872, accum_checksum:1991013568, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774177887661, freeze_ts:1713774177887661, submit_ts:1713774177887674, flushed_ts:1713774177893377, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:5703}, fs_cb_begin_ts=1713774182065829, log_life_time=4178168) [2024-04-22 16:23:02.065922] INFO [PALF] config_change_update_match_lsn_map (log_sliding_window.cpp:4048) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=32] config_change_update_match_lsn_map_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", added_memberlist=0, removed_memberlist=0) [2024-04-22 16:23:02.065933] INFO [PALF] append_config_meta_ (log_config_mgr.cpp:1586) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] append_config_meta_ success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", curr_proposal_id=7, args={server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, new_config_info={version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, log_ms_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:02.065995] INFO [PALF] change_config_ (log_config_mgr.cpp:865) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=57] append_config_meta success, start config change(ret=0, palf_id=1, self="172.23.88.129:2882", curr_proposal_id=7, args={server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}) [2024-04-22 16:23:02.066012] INFO [PALF] reconfirm (log_reconfirm.cpp:584) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] [PALF_EVENT] Reconfirm come into START_WORKING state(palf_id=1, new_proposal_id_=7, majority_max_log_server_="172.23.88.129:2882", majority_max_lsn_={lsn:356581}, majority_match_lsn={lsn:18446744073709551615}, last_lsn={lsn:356459}, last_log_proposal_id=6, last_end_lsn={lsn:356581}) [2024-04-22 16:23:02.065939] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1172) [10870][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=159] succ to submit log task to replay service(to_submit_lsn={lsn:356515}, to_submit_scn={val:1713774177880657607, v:0}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774182065832, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:356581}, next_to_submit_scn_:{val:1713774177880657608, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:356459}, end_lsn:{lsn:356581}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774177880657607, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774177880657607, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:1991013568, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774177880657607, v:0}}}}}) [2024-04-22 16:23:02.066039] INFO [PALF] serialize (log_meta.cpp:185) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] LogMeta serialize(*this={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, buf="", buf=0x7f3ce7dcd840, pos=779) [2024-04-22 16:23:02.066089] INFO [PALF] serialize_log_meta_ (log_engine.cpp:1528) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=47] serialize_log_meta_ success(ret=0, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_entry={header:{magic:19533, version:1, data_len:4072, data_checksum:861189549, header_checksum:4064556573}, buf_:0x7f3ce7dcd840}, log_meta_entry_header_len=24) [2024-04-22 16:23:02.066131] INFO [LIB] stat (utility.h:1165) [10867][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=83] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=120, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:02.066136] INFO [PALF] submit_io_task (log_io_worker.cpp:149) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=44] submit flush meta task success(io_task={LogIOTask:{BaseClass:"LogIOTask", palf_id:1, palf_epoch:1, create_task_ts:1713774182066027, push_cb_into_cb_pool_ts:-1, submit_seq_:28}, flush_meta_cb_ctx:{type:"CHANGE_CONFIG_META", proposal_id:7, config_version:{proposal_id:7, config_seq:8}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}}) [2024-04-22 16:23:02.066147] INFO [LIB] stat (utility.h:1165) [10867][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=15] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=9, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:02.066165] INFO [PALF] submit_flush_meta_task_ (log_engine.cpp:1287) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20] submit_flush_meta_task_ success(flush_meta_cb_ctx={type:"CHANGE_CONFIG_META", proposal_id:7, config_version:{proposal_id:7, config_seq:8}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}) [2024-04-22 16:23:02.066212] INFO [PALF] submit_flush_change_config_meta_task (log_engine.cpp:416) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=47] submit_flush_change_config_meta_task success(ret=0, flush_meta_cb_ctx={type:"CHANGE_CONFIG_META", proposal_id:7, config_version:{proposal_id:7, config_seq:8}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:02.066294] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.066307] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7, meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.066368] INFO [PALF] submit_change_config_meta_req_ (log_engine.h:298) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=36] submit_change_config_meta_req_ success(ret=0, member_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, msg_proposal_id=7, prev_log_proposal_id=6, prev_lsn={lsn:356459}, prev_mode_pid=7, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:02.066417] INFO [PALF] submit_config_log_ (log_config_mgr.cpp:2010) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=48] submit_config_log success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", dst_member_list=2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, proposal_id=7, prev_log_proposal_id=6, prev_lsn={lsn:356459}, prev_mode_pid=7, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:02.066482] INFO [PALF] change_config_ (log_config_mgr.cpp:891) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=64] submit_config_log success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:02.066491] WDIAG [PALF] reconfirm (log_reconfirm.cpp:618) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] wait start work log majority or log slide(palf_id=1, new_proposal_id_=7, majority_max_log_server_="172.23.88.129:2882", majority_max_lsn_={lsn:356581}, majority_max_accept_pid_=6) [2024-04-22 16:23:02.066498] INFO [PALF] reconfirm (log_reconfirm.cpp:638) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] reconfirm waiting retry(ret=-4023, palf_id=1, state=6) [2024-04-22 16:23:02.066505] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6] switch_state(ret=-4023, palf_id=1, old_role=1, role=1, old_state=3, state=3, old_leader="172.23.88.129:2882", leader="172.23.88.129:2882", old_leader_epoch=10, leader_epoch=10, need_next_loop=false, diff=1014) [2024-04-22 16:23:02.068989] INFO [PALF] append_meta (log_storage.cpp:269) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=37] LogStorage append meta success(ret=0, log_block_header_={magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, log_tail_={lsn:102400}, this={log_tail:{lsn:102400}, readable_log_tail:{lsn:102400}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67002368, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}) [2024-04-22 16:23:02.069054] INFO [PALF] inner_after_flush_meta (palf_handle_impl.cpp:4174) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=33] inner_after_flush_meta(flush_meta_cb_ctx={type:"CHANGE_CONFIG_META", proposal_id:7, config_version:{proposal_id:7, config_seq:8}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}) [2024-04-22 16:23:02.069100] INFO [PALF] after_flush_config_log (log_config_mgr.cpp:702) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=38] after_flush_config_log success(palf_id=1, self="172.23.88.129:2882", config_version={proposal_id:7, config_seq:8}, persistent_config_version={proposal_id:7, config_seq:8}) [2024-04-22 16:23:02.069118] INFO [PALF] ack_config_log (log_config_mgr.cpp:2090) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] ack_config_log success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", config_version={proposal_id:7, config_seq:8}, sender="172.23.88.129:2882", is_majority=false, ms_ack_list=1"172.23.88.129:2882";, alive_paxos_replica_num_=3) [2024-04-22 16:23:02.072480] INFO [PALF] ack_config_log (log_config_mgr.cpp:2090) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D92-0-0] [lt=140] ack_config_log success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", config_version={proposal_id:7, config_seq:8}, sender="172.23.88.131:2882", is_majority=true, ms_ack_list=2"172.23.88.129:2882";"172.23.88.131:2882";, alive_paxos_replica_num_=3) [2024-04-22 16:23:02.072510] INFO [PALF] ack_config_log (palf_handle_impl.cpp:4098) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D92-0-0] [lt=29] ack_config_log success(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.131:2882", msg_proposal_id=7, config_version={proposal_id:7, config_seq:8}) [2024-04-22 16:23:02.072538] INFO [PALF] config_change_update_match_lsn_map (log_sliding_window.cpp:4048) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D92-0-0] [lt=16] config_change_update_match_lsn_map_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", added_memberlist=0, removed_memberlist=0) [2024-04-22 16:23:02.164586] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:02.165568] INFO [PALF] config_change_update_match_lsn_map (log_sliding_window.cpp:4048) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17] config_change_update_match_lsn_map_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", added_memberlist=0, removed_memberlist=0) [2024-04-22 16:23:02.165603] INFO [PALF] confirm_start_working_log (log_config_mgr.cpp:1883) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=34] confirm_start_working_log success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", log_ms_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:02.165645] INFO [PALF] reconfirm (log_reconfirm.cpp:612) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=39] [PALF_EVENT] Reconfirm come into FINISHED state(palf_id=1, self="172.23.88.129:2882", saved_end_lsn={lsn:356581}) [2024-04-22 16:23:02.165653] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:02.165659] INFO [PALF] reconfirm_to_leader_active_ (log_state_mgr.cpp:655) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] [PALF_EVENT] reconfirm_to_leader_active begin(palf_id=1, self="172.23.88.129:2882", reconfirm_stage_cost=400327) [2024-04-22 16:23:02.165672] INFO [PALF] is_all_log_flushed_ (log_sliding_window.cpp:2343) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] is_all_log_flushed_(bool_ret=true, palf_id=1, self="172.23.88.129:2882", curr_end_lsn={lsn:356581}, max_flushed_end_lsn={lsn:356581}) [2024-04-22 16:23:02.165667] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:02.165680] INFO [PALF] clean_log_ (log_sliding_window.cpp:2514) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6] record last slide log info(ret=0, last_slide_log_id=2925, last_slide_scn={val:1713774177880657607, v:0}, last_slide_end_lsn={lsn:356581}, palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:02.165681] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177680355699, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774182165646) [2024-04-22 16:23:02.165697] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774181965574, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:02.165699] INFO [PALF] truncate (lsn_allocator.cpp:93) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11] truncate success(lsn={lsn:356581}, log_id=2925, scn={val:1713774177880657607, v:0}) [2024-04-22 16:23:02.165709] INFO [PALF] truncate_lsn_allocator_ (log_sliding_window.cpp:4366) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] lsn_allocator_.truncate success(ret=0, palf_id=1, self="172.23.88.129:2882", last_lsn={lsn:356581}, last_log_id=2925, last_scn={val:1713774177880657607, v:0}) [2024-04-22 16:23:02.165718] INFO [PALF] clean_log_ (log_sliding_window.cpp:2576) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] clean log finished(ret=0, palf_id=1, self="172.23.88.129:2882", max_log_id=2925, first_empty_log_id=-1, last_submit_log_id=2925, start_log_id=2926, max_log_id=2925, new_last_log_id=2925, new_last_scn={val:1713774177880657607, v:0}, new_last_log_end_lsn={lsn:356581}) [2024-04-22 16:23:02.165725] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=4284553, log_service_ts=1713774177880657, min_tx_service_ts=4611686018427387, timestamp={val:1713774177880657607, v:0}) [2024-04-22 16:23:02.165728] INFO [PALF] to_leader (log_group_buffer.cpp:353) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] to_leader finished(ret=0, available_buffer_size=33554432, reserved_buffer_size=41943040) [2024-04-22 16:23:02.165741] INFO [PALF] inc_update_scn_base (lsn_allocator.cpp:191) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] inc_update_scn_base success(scn_base=1713774177880657607, scn_delta=0, scn=1) [2024-04-22 16:23:02.165737] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774148194661603, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:02.165762] INFO [PALF] reset_match_lsn_map_ (log_sliding_window.cpp:3987) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15] reset_match_lsn_map_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", member_list=3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, max_flushed_end_lsn={lsn:356581}, committed_end_lsn={lsn:356581}) [2024-04-22 16:23:02.165783] INFO [PALF] to_leader_active (log_sliding_window.cpp:2628) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] to_leader_active success(ret=0, palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:02.165796] INFO [CLOG] insert (ob_role_change_service.cpp:98) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] insert event into set success(ret=0, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, free_idx=0) [2024-04-22 16:23:02.165826] INFO [CLOG] submit_role_change_event_ (ob_role_change_service.cpp:271) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] submit_role_change_event_ success(ret=0, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:02.165834] INFO [CLOG] on_role_change (ob_role_change_service.cpp:243) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] on_role_change success(ret=0, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:02.165840] INFO [PALF] to_leader_active_ (log_state_mgr.cpp:730) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5] to_leader_active_ finished!(ret=0, palf_id=1) [2024-04-22 16:23:02.165845] INFO [PALF] submit_broadcast_leader_info (log_config_mgr.cpp:593) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=4] submit_prepare_meta_req success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", proposal_id=7) [2024-04-22 16:23:02.165843] INFO [CLOG] handle (ob_role_change_service.cpp:217) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6] begin handle_role_change_event_(sequence:=1713774182165841, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:02.165852] INFO [PALF] reconfirm_to_leader_active_ (log_state_mgr.cpp:678) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] [PALF_EVENT] reconfirm_to_leader_active end(palf_id=1, ret=0, self="172.23.88.129:2882", reconfirm_to_active_cost=400520, role=1, state=2) [2024-04-22 16:23:02.165862] INFO [CLOG] remove (ob_role_change_service.cpp:117) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=18] remove slog from set success(ret=0, delete_idx=0, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:02.165896] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11] task handle constructed(*this={this:0x7f3ce7dcf510, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:02.165931] INFO [SHARE] add_event_with_retry (ob_event_history_table_operator.h:374) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] add_event_with_retry success(ret=0, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882)) [2024-04-22 16:23:02.165940] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] switch_state(ret=0, palf_id=1, old_role=1, role=1, old_state=3, state=2, old_leader="172.23.88.129:2882", leader="172.23.88.129:2882", old_leader_epoch=10, leader_epoch=10, need_next_loop=false, diff=384) [2024-04-22 16:23:02.165990] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.166003] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7, meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.165983] INFO [CLOG] handle_role_change_event_ (ob_role_change_service.cpp:345) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=9] begin log handler role change(curr_access_mode=1, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:5, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:02.166059] INFO [CLOG] is_replay_done (ob_replay_status.cpp:887) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=61] log stream finished replay(ls_id_={id:1}, min_unreplayed_lsn={lsn:356581}, end_lsn={lsn:356581}) [2024-04-22 16:23:02.166037] INFO [PALF] submit_change_config_meta_req_ (log_engine.h:298) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=32] submit_change_config_meta_req_ success(ret=0, member_list={learner_num:1, learner_array:[{server:"172.23.88.130:2882", timestamp:1, flag:0}]}, msg_proposal_id=7, prev_log_proposal_id=6, prev_lsn={lsn:356459}, prev_mode_pid=7, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:02.166071] INFO [CLOG] wait_replay_service_replay_done_ (ob_role_change_service.cpp:873) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=8] wait_replay_service_replay_done_ finish(ret=0, ls_id={id:1}, end_lsn={lsn:356581}, is_done=true) [2024-04-22 16:23:02.166089] INFO [CLOG] switch_to_leader (ob_log_apply_service.cpp:575) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] apply status switch_to_leader success(this={ls_id_:{id:1}, role_:1, proposal_id_:7, palf_committed_end_lsn_:{lsn:338140}, palf_committed_end_scn_:{val:1713774148155695009, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713774148155695009, v:0}}) [2024-04-22 16:23:02.166105] INFO [CLOG] switch_to_leader (ob_log_apply_service.cpp:1231) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=15] apply service switch_to_leader success(id={id:1}, proposal_id=7) [2024-04-22 16:23:02.166121] INFO [CLOG] switch_to_leader (ob_replay_status.cpp:809) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] replay status switch_to_leader(this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774182065832, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:356581}, next_to_submit_scn_:{val:1713774177880657608, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:356459}, end_lsn:{lsn:356581}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774177880657607, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774177880657607, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:1991013568, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774177880657607, v:0}}}}}) [2024-04-22 16:23:02.166165] INFO [CLOG] wait_replay_service_submit_task_clear_ (ob_role_change_service.cpp:892) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=37] wait_replay_service_submit_task_clear_ finish(ls_id={id:1}, is_clear=true) [2024-04-22 16:23:02.166196] INFO [STORAGE.TRANS] switch_state (ob_trans_ctx_mgr_v4.cpp:365) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] ObLSTxCtxMgr switch state success(ls_id=1, F_WORKING ~> T_PENDING, op=LEADER_TAKEOVER) [2024-04-22 16:23:02.166292] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.166319] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, curr_lsn:{lsn:356581}, write_buf:{total_size:168, count:1, write_buf:[{buf_:0x7f3cce25b115, buf_len:168}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.166374] INFO [LIB] stat (utility.h:1165) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=27, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:02.166396] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="LEADER", total_group_log_cnt=4, avg_log_batch_cnt=0, total_group_log_size=534, avg_group_log_size=133) [2024-04-22 16:23:02.166413] INFO [PALF] submit_log (palf_handle_impl.cpp:453) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, append size=18563) [2024-04-22 16:23:02.166425] INFO [LIB] stat (utility.h:1165) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT APPEND COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=197, this=0x7f3cd2d6d1d0, extra_info="id:1") [2024-04-22 16:23:02.166466] INFO [STORAGE.TRANS] submit_ls_log_ (ob_tx_ls_log_writer.h:231) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=30] [TxLsLogWriter] submit ls log success(ret=0, T::LOG_TYPE=1048576, ls_id_={id:1}) [2024-04-22 16:23:02.166484] INFO [STORAGE.TRANS] switch_to_leader (ob_trans_ctx_mgr_v4.cpp:844) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=13] [LsTxCtxMgr] switch_to_leader(ret=0, this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"T_PENDING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}) [2024-04-22 16:23:02.166501] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=14] follower to leader, current role change handler is(cursor=1, name="TRANS_SERVICE") [2024-04-22 16:23:02.166508] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6] follower to leader, current role change handler is(cursor=3, name="STORAGE_SCHEMA") [2024-04-22 16:23:02.166512] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=4, name="TABLET_SEQ_SYNC") [2024-04-22 16:23:02.166526] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=5] follower to leader, current role change handler is(cursor=5, name="DDL") [2024-04-22 16:23:02.166531] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=6, name="KEEP_ALIVE") [2024-04-22 16:23:02.166545] INFO [STORAGE.TRANS] switch_to_leader (ob_timestamp_service.cpp:261) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=8] ObTimestampService switch to leader success(ret=0, version={val:1713774177880657608, v:0}, last_id_=1713774187210958129, limited_id_=1713774187210958129, service_type=1) [2024-04-22 16:23:02.166552] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=7] follower to leader, current role change handler is(cursor=7, name="TIMESTAMP") [2024-04-22 16:23:02.166558] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=8, name="TRANS_ID") [2024-04-22 16:23:02.166563] INFO [CLOG] switch_to_leader (ob_garbage_collector.cpp:654) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=3] gc_handler switch_to_leader [2024-04-22 16:23:02.166567] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=9, name="GC_LS") [2024-04-22 16:23:02.166586] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=7] follower to leader, current role change handler is(cursor=10, name="MAJOR_FREEZE") [2024-04-22 16:23:02.166591] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=3] follower to leader, current role change handler is(cursor=14, name="GAIS") [2024-04-22 16:23:02.166597] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6] follower to leader, current role change handler is(cursor=15, name="DAS_ID") [2024-04-22 16:23:02.166677] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=16, name="RESTORE_SERVICE") [2024-04-22 16:23:02.166689] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=11] follower to leader, current role change handler is(cursor=17, name="RESERVED_SNAPSHOT") [2024-04-22 16:23:02.166699] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] follower to leader, current role change handler is(cursor=18, name="MEDIUM_COMPACTION") [2024-04-22 16:23:02.166705] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=23, name="NET_ENDPOINT_EGRESS") [2024-04-22 16:23:02.166747] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=24, name="HEARTBEAT_SERVICE") [2024-04-22 16:23:02.166766] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC60-0-0] [lt=26][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.166790] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC60-0-0] [lt=13] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:02.166773] INFO [STORAGE.DUP_TABLE] leader_takeover_ (ob_dup_table_util.cpp:1443) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=3] Leader Takeover(ret=0, is_resume=false, this={ls_id_:{id:1}, ls_state_helper_:{module_name:"DupTableLSHandler", cur_state_:{role_state_:1, offline_state_:21, stop_state_:31}}, dup_ls_ckpt_:{dup_ls_meta_:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}, lease_log_rec_scn_:{val:18446744073709551615, v:3}, start_replay_scn_:{val:18446744073709551615, v:3}, readable_ckpt_base_scn_is_accurate_:true}, lease_mgr_ptr_:null, tablets_mgr_ptr_:null, ts_sync_mgr_ptr_:null, log_operator_:null, interface_stat_:{dup_table_follower_read_succ_cnt_:0, dup_table_follower_read_tablet_not_exist_cnt_:0, dup_table_follower_read_tablet_not_ready_cnt_:0, dup_table_follower_read_lease_expired_cnt_:0, dup_table_redo_sync_succ_cnt_:0, dup_table_redo_sync_fail_cnt_:0, dup_table_max_applying_scn_:{val:0, v:0}, dup_table_log_entry_cnt_:0, dup_table_log_entry_total_size_:0, dup_table_log_replay_total_time_:0, dup_table_log_deser_total_time_:0, dup_table_lease_log_sync_total_time_:0, dup_table_tablet_log_sync_total_time_:0}}) [2024-04-22 16:23:02.166816] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=42] follower to leader, current role change handler is(cursor=26, name="DUP_TABLE") [2024-04-22 16:23:02.166826] INFO [STORAGE.TABLELOCK] switch_to_leader (ob_lock_table.cpp:726) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] start to check and clear obj lock when switch to leader(ret=0, parent_->get_ls_id()={id:1}) [2024-04-22 16:23:02.166863] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6] follower to leader, current role change handler is(cursor=27, name="OBJ_LOCK_GARBAGE_COLLECT_SERVICE") [2024-04-22 16:23:02.166802] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC60-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.166914] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=7] follower to leader, current role change handler is(cursor=31, name="BACKUP_TASK_SCHEDULER") [2024-04-22 16:23:02.166943] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=32, name="BACKUP_DATA_SERVICE") [2024-04-22 16:23:02.166976] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] follower to leader, current role change handler is(cursor=33, name="BACKUP_CLEAN_SERVICE") [2024-04-22 16:23:02.166999] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=3] follower to leader, current role change handler is(cursor=34, name="BACKUP_ARCHIVE_SERVICE") [2024-04-22 16:23:02.167021] INFO [STORAGE] switch_to_leader (ob_transfer_handler.cpp:288) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6] [TRANSFER]switch to leader finish [2024-04-22 16:23:02.167031] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] follower to leader, current role change handler is(cursor=35, name="TRANSFER_HANDLER") [2024-04-22 16:23:02.167068] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6] follower to leader, current role change handler is(cursor=36, name="COMMON_LS_SERVICE") [2024-04-22 16:23:02.167094] WDIAG [STORAGE.TRANS] switch_state (ob_trans_ctx_mgr_v4.cpp:368) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4109] ObLSTxCtxMgr switch state failed(ret=-4109, ls_id=1, state=T_PENDING, op=UNBLOCK_NORMAL) [2024-04-22 16:23:02.167102] WDIAG [STORAGE.TRANS] unblock_normal (ob_trans_ctx_mgr_v4.cpp:1102) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4109] switch state error(ret=-4109, ret="OB_STATE_NOT_MATCH", manager={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"T_PENDING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}) [2024-04-22 16:23:02.167120] INFO [STORAGE.TRANS] unblock_normal (ob_trans_ctx_mgr_v4.cpp:1104) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=18] unblock ls normally(ret=-4109, manager={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"T_PENDING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}) [2024-04-22 16:23:02.167113] INFO [STORAGE] scheduler_transfer_handler_ (ob_transfer_service.cpp:200) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBD-0-0] [lt=17] start do transfer handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:02.167145] WDIAG [STORAGE.TRANS] unblock_normal (ob_ls_tx_service.cpp:843) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4109] ls tx service unblock normal failed(ret=-4109, ls_id={id:1}) [2024-04-22 16:23:02.167150] INFO [STORAGE] switch_to_leader (ob_ls_block_tx_service.cpp:73) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=5] [BLOCK_TX]switch to leader finish [2024-04-22 16:23:02.167156] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6] follower to leader, current role change handler is(cursor=37, name="BLOCK_TX_SERVICE") [2024-04-22 16:23:02.167180] WDIAG [SHARE] get_tenant_primary_zone_score (ob_primary_zone_util.cpp:654) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC72-0-0] [lt=12][errcode=-4016] zone list count unexpected(ret=-4016) [2024-04-22 16:23:02.167173] INFO [WR] inner_switch_to_leader (ob_wr_service.cpp:100) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=5] current observer is leader, start to dispatch workload repository snapshot timer(this={ObIRoleChangeSubHandler:{ObIRoleChangeSubHandler:"Dummy"}, is_inited:true, wr_timer_task:{ObTimerTask:tasktype:N9oceanbase5share22WorkloadRepositoryTaskE, snapshot_interval:60, tg_id:290, timeout_ts:0, is_running_task:false}}) [2024-04-22 16:23:02.167193] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=19] follower to leader, current role change handler is(cursor=38, name="WORKLOAD_REPOSITORY_SERVICE") [2024-04-22 16:23:02.167189] WDIAG [SHARE] get_tenant_primary_zone_array (ob_primary_zone_util.cpp:613) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC72-0-0] [lt=7][errcode=-4016] failed to get tenant primary zone score(ret=-4016, ret="OB_ERR_UNEXPECTED", tenant_schema={tenant_id:1, schema_version:1, tenant_name:"sys", zone_list:[cnt:0], primary_zone:"", charset_type:2, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:"", zone_replica_attr_array:[cnt:0], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:-1, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2024-04-22 16:23:02.167226] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] follower to leader, current role change handler is(cursor=42, name="CLONE_SCHEDULER") [2024-04-22 16:23:02.167252] INFO [CLOG] switch_to_leader (ob_role_change_handler.cpp:111) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=7] follower to leader, current role change handler is(cursor=43, name="MVIEW_MAINTENANCE_SERVICE_LOG_BASE_TYPE") [2024-04-22 16:23:02.167256] INFO [CLOG] switch_follower_to_leader_ (ob_role_change_service.cpp:622) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=4] switch_follower_to_leader_ success(ret=0, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:02.167303] INFO [CLOG] handle_role_change_event_ (ob_role_change_service.cpp:350) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=43] end log handler role change(ret=0, curr_access_mode=1, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}, retry_ctx={reason:0}) [2024-04-22 16:23:02.167351] INFO [CLOG] handle_role_change_event_ (ob_role_change_service.cpp:351) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=47] begin restore handler role change(curr_access_mode=1, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:02.167389] INFO [CLOG] handle_role_change_event_ (ob_role_change_service.cpp:356) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=36] end restore handler role change(ret=0, curr_access_mode=1, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:02.167457] INFO [STORAGE] report_replica_info (ob_ls.cpp:1419) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=38] submit log stream update task(ls_meta_.tenant_id_=1, ls_meta_.ls_id_={id:1}) [2024-04-22 16:23:02.167464] INFO [CLOG] handle (ob_role_change_service.cpp:223) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=7] end handle_role_change_event_(sequence:=1713774182165841, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:02.167682] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2E-0-0] [lt=16][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.167701] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2E-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:02.167711] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2E-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.168066] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10806][T1_CloneSche][T1][YB42AC175881-000616AB09AFEBB2-0-0] [lt=5][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.168087] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10806][T1_CloneSche][T1][YB42AC175881-000616AB09AFEBB2-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_clone_job, table_name.ptr()="data_size:15, data:5F5F616C6C5F636C6F6E655F6A6F62", ret=-5019) [2024-04-22 16:23:02.168098] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10806][T1_CloneSche][T1][YB42AC175881-000616AB09AFEBB2-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.168141] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=20][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.168179] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=31] table not exist(tenant_id=1, database_id=201001, table_name=__all_transfer_task, table_name.ptr()="data_size:19, data:5F5F616C6C5F7472616E736665725F7461736B", ret=-5019) [2024-04-22 16:23:02.168190] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.168322] INFO [PALF] get_log (palf_handle_impl.cpp:3994) [11430][T1_L0_G1][T1][YB42AC175881-000616AB08FFEBB4-0-0] [lt=8] submit fetch log task success(ret=0, palf_id=1, server="172.23.88.131:2882", fetch_type=0, prev_lsn={lsn:356581}, start_lsn={lsn:356749}, fetch_log_size=41943040, fetch_log_count=2047, accepted_mode_pid=7) [2024-04-22 16:23:02.168338] INFO [PALF] handle (fetch_log_engine.cpp:284) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=16] handle fetch_log_task(fetch_log_task={timestamp_us:1713774182168310, id:1, server:"172.23.88.131:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:7, prev_lsn:{lsn:356581}, start_lsn:{lsn:356749}, log_size:41943040, log_count:2047, accepted_mode_pid:7}) [2024-04-22 16:23:02.168362] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3583) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=20] no need fetch_log_from_storage(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.131:2882", fetch_start_lsn={lsn:356749}, prev_lsn={lsn:356581}, max_flushed_end_lsn={lsn:356581}, access_mode=1) [2024-04-22 16:23:02.168375] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3705) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=12] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, prev_lsn={lsn:356581}, iterator={iterator_impl:{buf_:null, next_round_pread_size:0, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:NULL, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogGroupEntryHeader:{magic:0, version:0, group_size:0, proposal_id:9223372036854775807, committed_lsn:{lsn:18446744073709551615}, max_scn:{val:18446744073709551615, v:3}, accumulated_checksum:0, log_id:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}, fetch_log_count=2047, has_fetched_log_count=0) [2024-04-22 16:23:02.168404] WDIAG [PALF] try_send_committed_info_ (palf_handle_impl.cpp:3507) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4002] invalid arguments(ret=-4002, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.131:2882", log_lsn={lsn:356581}, log_end_lsn={lsn:356581}, log_proposal_id=9223372036854775807) [2024-04-22 16:23:02.168458] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=9] submit_change_mode_meta_req success(ret=0, palf_id=1, server="172.23.88.131:2882", mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=7) [2024-04-22 16:23:02.168471] INFO [PALF] handle (fetch_log_engine.cpp:321) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=11] handle fetch log task(ret=0, palf_id=1, handle_cost_time_us=134, fetch_log_task={timestamp_us:1713774182168310, id:1, server:"172.23.88.131:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:7, prev_lsn:{lsn:356581}, start_lsn:{lsn:356749}, log_size:41943040, log_count:2047, accepted_mode_pid:7}, fetch_stat={total_size:0, group_log_cnt:0, read_cost:0, get_cost:0, send_cost:0}) [2024-04-22 16:23:02.168936] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.168956] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=15] table not exist(tenant_id=1, database_id=201001, table_name=__all_transfer_task, table_name.ptr()="data_size:19, data:5F5F616C6C5F7472616E736665725F7461736B", ret=-5019) [2024-04-22 16:23:02.169241] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:229) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=0] statistics the time that get needed to backfill tablets(ls id={id:1}, cost_time=136) [2024-04-22 16:23:02.169276] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2181) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=23] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:02.169336] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:235) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=53] [TRANSFER] transfer handler process(ret=0, ls_id={id:1}) [2024-04-22 16:23:02.169356] INFO [STORAGE] scheduler_transfer_handler_ (ob_transfer_service.cpp:200) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBBF-0-0] [lt=6] start do transfer handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:02.169950] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBC1-0-0] [lt=6][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.169964] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBC1-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_transfer_task, table_name.ptr()="data_size:19, data:5F5F616C6C5F7472616E736665725F7461736B", ret=-5019) [2024-04-22 16:23:02.170078] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=4676, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:02.170093] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=900) [2024-04-22 16:23:02.170142] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=78, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:02.170603] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBC1-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.170634] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBC1-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_transfer_task, table_name.ptr()="data_size:19, data:5F5F616C6C5F7472616E736665725F7461736B", ret=-5019) [2024-04-22 16:23:02.170875] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:229) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBC1-0-0] [lt=0] statistics the time that get needed to backfill tablets(ls id={id:1}, cost_time=113) [2024-04-22 16:23:02.170894] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2181) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBC1-0-0] [lt=17] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774187210958129, latest_log_ts:{val:1713774170544377163, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:02.170941] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:235) [10860][T1_TransferServ][T1][YB42AC175881-000616AB084FEBC1-0-0] [lt=46] [TRANSFER] transfer handler process(ret=0, ls_id={id:1}) [2024-04-22 16:23:02.172075] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D93-0-0] [lt=29][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.172110] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D93-0-0] [lt=30][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2926, prev_log_proposal_id:7, committed_end_lsn:{lsn:356749}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.173730] INFO [STORAGE.TRANS] switch_state (ob_trans_ctx_mgr_v4.cpp:365) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=12] ObLSTxCtxMgr switch state success(ls_id=1, T_PENDING ~> L_WORKING, op=SWL_CB_SUCC) [2024-04-22 16:23:02.173799] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.173811] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4122] post gts request failed(ret=-4122, ret="OB_RPC_POST_ERROR", server="172.23.88.130:2882", msg={tenant_id:1, srr:[mts=1713774182173784], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:02.173825] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4122] post gts request failed(ret=-4122, ret="OB_RPC_POST_ERROR", leader="172.23.88.130:2882", msg={tenant_id:1, srr:[mts=1713774182173784], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:02.173854] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774182173784]}) [2024-04-22 16:23:02.173870] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] query gts fail(tmp_ret=-4122, leader="172.23.88.130:2882") [2024-04-22 16:23:02.173879] INFO [STORAGE.TRANS] try_wait_gts_and_inc_max_commit_ts_ (ob_trans_ctx_mgr_v4.cpp:820) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=8] try wait gts(ret=-4038, ret="OB_NOT_MASTER", ls_id={id:1}, tenant_id=1, max_replay_commit_version={val:1, v:0}, gts={val:18446744073709551615, v:3}) [2024-04-22 16:23:02.173894] INFO [STORAGE.TRANS] on_start_working_log_cb_succ (ob_trans_ctx_mgr_v4.cpp:726) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=12] [LsTxCtxMgr Role Change] on_start_working_log_cb_succ(ret=0, this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"L_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}) [2024-04-22 16:23:02.173914] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=14] [APPLY STAT CB APPEND COST TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=5044, this=0x7f3cd2df8280, extra_info="") [2024-04-22 16:23:02.173921] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=6442, this=0x7f3cd2df84b0, extra_info="") [2024-04-22 16:23:02.173930] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=0, this=0x7f3cd2df86e0, extra_info="") [2024-04-22 16:23:02.173934] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=4] [APPLY STAT CB EXECUTE TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=-423, this=0x7f3cd2df8910, extra_info="") [2024-04-22 16:23:02.173939] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=5] [APPLY STAT CB TOTAL TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=11064, this=0x7f3cd2df8b40, extra_info="") [2024-04-22 16:23:02.189937] INFO [SHARE] run2 (ob_async_task_queue.cpp:109) [10495][RSAsyncTask2][T0][Y0-000616AB0B0FEBDB-0-0] [lt=16] async task queue start [2024-04-22 16:23:02.190052] INFO [SHARE] run2 (ob_async_task_queue.cpp:109) [10494][RSAsyncTask1][T0][Y0-000616AB0A0FEC78-0-0] [lt=30] async task queue start [2024-04-22 16:23:02.190039] INFO [SHARE] run2 (ob_async_task_queue.cpp:109) [10493][RSAsyncTask0][T0][Y0-000616AB09FFEBE6-0-0] [lt=11] async task queue start [2024-04-22 16:23:02.190079] INFO [SHARE] run2 (ob_async_task_queue.cpp:109) [10496][RSAsyncTask3][T0][Y0-000616AB0A1FEC2C-0-0] [lt=15] async task queue start [2024-04-22 16:23:02.189995] INFO [SHARE] run2 (ob_async_task_queue.cpp:109) [10498][RSInspectTask][T0][Y0-0000000000000000-0-0] [lt=143] async task queue start [2024-04-22 16:23:02.190468] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBC-0-0] [lt=27][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.190498] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBC-0-0] [lt=22] table not exist(tenant_id=1, database_id=201001, table_name=__all_rootservice_event_history, table_name.ptr()="data_size:31, data:5F5F616C6C5F726F6F74736572766963655F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:02.190511] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBC-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.191556] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [10492][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=36] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-04-22 16:23:02.191593] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [10492][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=21] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService13ObRestartTaskE) [2024-04-22 16:23:02.191651] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:355) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2024-04-22 16:23:02.191663] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=12] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1713772869) [2024-04-22 16:23:02.191670] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:311) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=6][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2024-04-22 16:23:02.191723] INFO [SHARE.PT] update_leader_replica_ (ob_inmemory_ls_table.cpp:230) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=5] add leader replica for sys ls(new_replica={modify_time_us:1713774182191721, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info_={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1713774182191721, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:02.191792] INFO [SHARE.PT] update (ob_inmemory_ls_table.cpp:145) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=7] update ls replica(ret=0, ret="OB_SUCCESS", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1713774182191721, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:02.191846] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=34][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:02.192193] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBD-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.192218] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDD-0-0] [lt=7][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.192220] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBD-0-0] [lt=21] table not exist(tenant_id=1, database_id=201001, table_name=__all_rootservice_event_history, table_name.ptr()="data_size:31, data:5F5F616C6C5F726F6F74736572766963655F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:02.192236] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDD-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:02.192248] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDD-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.192339] INFO run1 (ob_timer.cpp:360) [11481][][T0][Y0-0000000000000000-0-0] [lt=0] timer thread started(this=0x7f3d1bcb39c0, tid=11481, lbt()=0x5d1c83c 0x59d6ad6 0x16c3d541 0x16c398f7 0x7f3d1c738ea5 0x7f3d1c461b0d) [2024-04-22 16:23:02.192734] INFO [SHARE.PT] update_follower_replica_ (ob_inmemory_ls_table.cpp:170) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=7] add follower replica for sys ls(new_replica={modify_time_us:1713774182192732, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2024-04-22 16:23:02.192758] INFO [SHARE.PT] update_follower_replica_ (ob_inmemory_ls_table.cpp:173) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=23] inmemory ls table add replica(ret=0, ret="OB_SUCCESS", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2024-04-22 16:23:02.192794] INFO [SHARE.PT] update (ob_inmemory_ls_table.cpp:145) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4] update ls replica(ret=0, ret="OB_SUCCESS", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1713774182191721, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774182192732, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:02.193538] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:02.193561] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=17][errcode=-4122] rpc task_execute fail(ret=-4122, tenant_id=1, svr="172.23.88.130:2882", timeout=29999908, timeout_timestamp=1713774212193406) [2024-04-22 16:23:02.193571] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:250) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=9][errcode=-4122] task execute failed(ret=-4122) [2024-04-22 16:23:02.193576] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] execute with sql failed(ret=-4122) [2024-04-22 16:23:02.193581] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] fail execute plan(ret=-4122) [2024-04-22 16:23:02.193586] WDIAG [SQL] open (ob_result_set.cpp:158) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] execute plan failed(ret=-4122) [2024-04-22 16:23:02.193613] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4122) [2024-04-22 16:23:02.193633] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=17][errcode=-4122] open result set failed(ret=-4122) [2024-04-22 16:23:02.193640] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=6][errcode=-4122] result set open failed(ret=-4122, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-04-22 16:23:02.193656] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=15][errcode=-4122] execute failed(ret=-4122, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:02.193664] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=7][errcode=-4122] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4122, err_:"OB_RPC_POST_ERROR", retry_type:0, client_ret:-4122}, need_retry=false) [2024-04-22 16:23:02.193679] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=8][errcode=-4122] result set close failed(ret=-4122) [2024-04-22 16:23:02.193684] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] result set close failed(ret=-4122) [2024-04-22 16:23:02.193687] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=3][errcode=-4122] failed to close result(close_ret=-4122, ret=-4122) [2024-04-22 16:23:02.193722] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4122, ret=-4122) [2024-04-22 16:23:02.193735] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=12][errcode=-4122] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, aret=-4122, ret=-4122) [2024-04-22 16:23:02.193758] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=22][errcode=-4122] execute sql failed(ret=-4122, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-04-22 16:23:02.193765] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=6][errcode=-4122] retry_while_no_tenant_resource failed(ret=-4122, tenant_id=1) [2024-04-22 16:23:02.193769] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] execute_read failed(ret=-4122, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:02.193778] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=8][errcode=-4122] query failed(ret=-4122, conn=0x7f3cd1498050, start=1713774182193406, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-04-22 16:23:02.193785] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=7][errcode=-4122] read failed(ret=-4122) [2024-04-22 16:23:02.193791] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] execute sql failed(ret=-4122, ret="OB_RPC_POST_ERROR", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-04-22 16:23:02.193829] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=5][errcode=-4122] load failed(ret=-4122, for_update=false) [2024-04-22 16:23:02.193835] WDIAG [SHARE] load_refresh_schema_status (ob_schema_status_proxy.cpp:240) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=5][errcode=-4122] fail to load core table(ret=-4122) [2024-04-22 16:23:02.193839] INFO [SHARE] load_refresh_schema_status (ob_schema_status_proxy.cpp:284) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4] [SCHEMA_STATUS] load refreshed schema status(ret=-4122) [2024-04-22 16:23:02.193878] WARN update_fail_count (ob_root_service.cpp:10758) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=6][errcode=-4752] rootservice start process has failure(msg="rootservice start()/do_restart() has failure", ret=-4122, ret="OB_RPC_POST_ERROR", fail_cnt=10) [2024-04-22 16:23:02.193944] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7A-0-0] [lt=4][errcode=-4122] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-4122) [2024-04-22 16:23:02.197631] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=47][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:02.197711] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=51][errcode=-4122] set config rpc failed(ret=-4122, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:02.198121] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=159] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774182197884), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -4122, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:02.198173] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=47][errcode=-4122] execute cmd failed(ret=-4122) [2024-04-22 16:23:02.198197] WDIAG [SQL] open (ob_result_set.cpp:158) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=22][errcode=-4122] execute plan failed(ret=-4122) [2024-04-22 16:23:02.198222] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=17][errcode=-4122] open result set failed(ret=-4122) [2024-04-22 16:23:02.198242] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=17][errcode=-4122] result set open failed(ret=-4122, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:02.198263] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=20][errcode=-4122] execute failed(ret=-4122, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:02.198285] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=20][errcode=-4122] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4122, err_:"OB_RPC_POST_ERROR", retry_type:0, client_ret:-4122}, need_retry=false) [2024-04-22 16:23:02.198312] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=22][errcode=-4122] result set close failed(ret=-4122) [2024-04-22 16:23:02.198323] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=10][errcode=-4122] result set close failed(ret=-4122) [2024-04-22 16:23:02.198333] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=9][errcode=-4122] failed to close result(close_ret=-4122, ret=-4122) [2024-04-22 16:23:02.198398] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBE8-0-0] [lt=45][errcode=-4122] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-4122, ret=-4122) [2024-04-22 16:23:02.198416] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10493][RSAsyncTask0][T0][YB42AC175881-000616AB09FFEBE8-0-0] [lt=15][errcode=-4122] failed to process final(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, aret=-4122, ret=-4122) [2024-04-22 16:23:02.198432] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10493][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4122] execute sql failed(ret=-4122, tenant_id=1, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:02.198526] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10493][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=52][errcode=-4122] retry_while_no_tenant_resource failed(ret=-4122, tenant_id=1) [2024-04-22 16:23:02.198541] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10493][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4122] execute_write failed(ret=-4122, tenant_id=1, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881', is_user_sql=false) [2024-04-22 16:23:02.198557] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10493][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] execute_write failed(ret=-4122, tenant_id=1, sql="ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'") [2024-04-22 16:23:02.198571] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10493][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4122] execute sql failed(ret=-4122, conn=0x7f3cd2268050, start=1713774182194073, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:02.198641] WDIAG [SHARE] store (ob_inner_config_root_addr.cpp:119) [10493][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] execute sql failed(ret=-4122, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:02.198655] WDIAG [SHARE] store (ob_rs_mgr.cpp:120) [10493][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] store rs list failed(agent=0, tmp_ret=-4122, addr_list=[{server:"172.23.88.129:2882", role:1, sql_port:2881}, {server:"172.23.88.131:2882", role:2, sql_port:2881}], force=true) [2024-04-22 16:23:02.198778] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10493][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] task process failed, start retry(max retry time=9223372036854775807, retry interval=1000000, ret=-4122) [2024-04-22 16:23:02.198852] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC6F-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.198884] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC6F-0-0] [lt=25] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:02.198897] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC6F-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.199448] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:02.235885] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:270) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=66][errcode=-4122] address in blacklist(ret=-4122, addr="172.23.88.130:2882") [2024-04-22 16:23:02.235922] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.235933] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={BASE:{msg_type:"Accept Request", id:1, sender:"172.23.88.129:2882", receiver:"172.23.88.130:2882", restart_counter:1, ballot_number:10, debug_ts:{src_construct_ts:"23:02.235843", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1713774182235843}, biggest_min_cluster_version_ever_seen:4.3.0.1}, lease_start_ts_on_proposer:1433056409, lease_interval:4000000, membership_version:{proposal_id:7, config_seq:8}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.236026] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04A-0-0] [lt=5][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:02.236048] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04A-0-0] [lt=21][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:02.236069] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04A-0-0] [lt=20][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:02.236093] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04A-0-0] [lt=23][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:02.237332] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04C-0-0] [lt=114][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:02.237365] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04C-0-0] [lt=32][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:02.237385] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04C-0-0] [lt=19][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:02.237398] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04C-0-0] [lt=13][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:02.237543] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04C-0-0] [lt=110][errcode=0] compare between invalid priority [2024-04-22 16:23:02.239759] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=6][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.239789] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:02.239804] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:02.239815] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:02.239824] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:02.239832] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:02.239838] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:02.239858] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=17][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:02.239863] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:02.239867] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:02.239874] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=7][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:02.239878] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:02.239883] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED77-0-0] [lt=4][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.261265] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:02.265733] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:02.265770] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177880657607, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774182265727) [2024-04-22 16:23:02.265783] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774182165704, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:02.265803] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:778) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=7, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:02.265842] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:358) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-04-22 16:23:02.265849] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] start TenantWeakReadClusterService(tenant_id=1) [2024-04-22 16:23:02.266500] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECED-0-0] [lt=4][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.266526] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECED-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-04-22 16:23:02.266539] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECED-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:02.266620] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:449) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1713774182266615, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=786, wlock_time=25, check_leader_time=2, query_version_time=0, persist_version_time=0) [2024-04-22 16:23:02.266650] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=7, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:02.266700] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=4385280, log_service_ts=1713774177880657, min_tx_service_ts=4611686018427387, timestamp={val:1713774177880657607, v:0}) [2024-04-22 16:23:02.266712] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=1, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774177773342916, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:02.282035] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.282070] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4122] post gts request failed(ret=-4122, ret="OB_RPC_POST_ERROR", server="172.23.88.130:2882", msg={tenant_id:1, srr:[mts=1713774182282021], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:02.282087] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4122] post gts request failed(ret=-4122, ret="OB_RPC_POST_ERROR", leader="172.23.88.130:2882", msg={tenant_id:1, srr:[mts=1713774182282021], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:02.282119] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774177775345], gts:1713774177773342916, latest_srr:[mts=1713774182282021]}) [2024-04-22 16:23:02.282130] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:175) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4122] refresh gts failed(ret=-4122, ret="OB_RPC_POST_ERROR", gts_tenant_info={v:1}) [2024-04-22 16:23:02.282140] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts functor(ret=-4122, ret="OB_RPC_POST_ERROR", gts_tenant_info={v:1}) [2024-04-22 16:23:02.282155] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFB81-0-0] [lt=25][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713774182282102}) [2024-04-22 16:23:02.297520] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:785, tid:10829}]) [2024-04-22 16:23:02.299154] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:181) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=73] dummy location not credible, need refresh [2024-04-22 16:23:02.299275] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:226) [11015][PxTargetMgr0][T1][Y0-0000000000000000-0-0] [lt=66] refresh locatition cache for target_monitor(tenant_id_=1, refresh_ctrl=11) [2024-04-22 16:23:02.299312] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] refresh dummy location cache(ret=0) [2024-04-22 16:23:02.299359] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:02.299539] WDIAG [SQL.ENG] query_statistics (ob_px_tenant_target_monitor.cpp:275) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=156][errcode=-4122] send rpc to query statistics failed, need send refresh all(ret=-4122) [2024-04-22 16:23:02.299570] WDIAG [SQL.ENG] refresh_statistics (ob_px_tenant_target_monitor.cpp:114) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4122] query statistics failed(ret=-4122) [2024-04-22 16:23:02.299587] WDIAG [COMMON] operator() (ob_px_target_mgr.h:158) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] target monitor refresh statistics failed(ret=-4122, px_tenant_info={v:1}, px_res_info->get_target_monitor()={is_init:true, tenant_id:1, server:"172.23.88.129:2882", dummy_cache_leader:"0.0.0.0:0", role:2}) [2024-04-22 16:23:02.322790] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.365283] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.365325] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:356581}, curr_lsn:{lsn:356749}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b1bd, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.365833] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=4484158, log_service_ts=1713774177880657, min_tx_service_ts=4611686018427387, timestamp={val:1713774177880657608, v:0}) [2024-04-22 16:23:02.365862] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=1, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774177773342916, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:02.369923] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D94-0-0] [lt=11][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.369945] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D94-0-0] [lt=16][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2927, prev_log_proposal_id:7, committed_end_lsn:{lsn:356871}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.399785] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:02.461429] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:02.465335] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.465365] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:356749}, curr_lsn:{lsn:356871}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b237, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.465880] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:02.465932] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177880657608, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774182465870) [2024-04-22 16:23:02.465952] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774182265797, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:02.465992] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=4584455, log_service_ts=1713774177880657, min_tx_service_ts=4611686018427387, timestamp={val:1713774177880657609, v:0}) [2024-04-22 16:23:02.466012] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=1, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774177773342916, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:02.471766] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D95-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.471801] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D95-0-0] [lt=28][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2928, prev_log_proposal_id:7, committed_end_lsn:{lsn:356993}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.494664] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [10835][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=65] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f3cec9dc2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-22 16:23:02.565406] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.565450] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:356871}, curr_lsn:{lsn:356993}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b2b1, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.565973] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] get wrs ts(ls_id={id:1}, delta=4684499, log_service_ts=1713774177880657, min_tx_service_ts=4611686018427387, timestamp={val:1713774177880657610, v:0}) [2024-04-22 16:23:02.566002] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=1, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774177773342916, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:02.570752] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D96-0-0] [lt=24][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.570775] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D96-0-0] [lt=20][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2929, prev_log_proposal_id:7, committed_end_lsn:{lsn:357115}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.600103] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:02.638453] INFO [COMMON] handle (memory_dump.cpp:468) [10428][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=44] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-04-22 16:23:02.643468] INFO handle (memory_dump.cpp:589) [10428][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 134 204 0 1 4 1 1 0 1 5 1 4 0 1 7 1 2 0 1 8 62 1 0 1 12 1 2 0 1 14 1 1 0 1 18 2 2 0 500 0 55 204 0 500 7 3 3 0 500 8 55 1 0 500 9 3 1 0 500 10 6 2 0 500 17 12 7 0 500 22 3 48 0 500 23 16 9 0 508 0 4 8 0 508 8 15 1 0 509 0 3 8 0 509 8 9 1 0 cost_time: 4903 [2024-04-22 16:23:02.661600] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=85] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:02.665500] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.665526] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:356993}, curr_lsn:{lsn:357115}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b32b, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.666019] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:02.666046] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177880657610, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774182666011) [2024-04-22 16:23:02.666058] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774182465963, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:02.666094] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=4784397, log_service_ts=1713774177880657, min_tx_service_ts=4611686018427387, timestamp={val:1713774177880657611, v:0}) [2024-04-22 16:23:02.666106] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=1, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774177773342916, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:02.668774] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.668811] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:02.668826] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, ret=-5019) [2024-04-22 16:23:02.668835] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=8][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_service_epoch, db_name=oceanbase) [2024-04-22 16:23:02.668850] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_service_epoch) [2024-04-22 16:23:02.668865] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:02.668870] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:02.668877] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_service_epoch' doesn't exist [2024-04-22 16:23:02.668883] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:02.668888] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:02.668893] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:02.668897] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:02.668903] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:02.668908] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:02.668913] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:02.668927] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:02.668935] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:02.668944] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=8][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:02.668949] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:02.668956] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' , ret=-5019) [2024-04-22 16:23:02.668963] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:02.668971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:02.668987] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:02.669020] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=30][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:02.669027] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:02.669032] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:02.669052] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:02.669058] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, aret=-5019, ret=-5019) [2024-04-22 16:23:02.669068] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:02.669074] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:02.669083] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=8][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:02.669090] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=6][errcode=-5019] query failed(ret=-5019, conn=0x7f3d1177a050, start=1713774182668646, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:02.669098] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=8][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:02.669104] WDIAG [SHARE] inner_get_service_epoch_ (ob_service_epoch_proxy.cpp:223) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED2F-0-0] [lt=4][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:02.674038] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D97-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.674067] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D97-0-0] [lt=24][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2930, prev_log_proposal_id:7, committed_end_lsn:{lsn:357237}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.706728] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.707104] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.707620] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.707652] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.707842] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.708133] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.708137] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.708665] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=61] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.708851] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.708986] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.710659] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.711714] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.711734] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.712449] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.712517] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.713033] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.713258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.713663] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.715095] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=12] gc stale ls task succ [2024-04-22 16:23:02.715524] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.716399] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10057, mem_used=76950528) [2024-04-22 16:23:02.717160] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:02.717687] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.717738] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.717776] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.717852] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:02.723413] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E8-0-0] [lt=4][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.723461] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E8-0-0] [lt=45][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:357115}, curr_lsn:{lsn:357237}, write_buf:{total_size:128, count:1, write_buf:[{buf_:0x7f3cce25b3a5, buf_len:128}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.723520] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:156) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E8-0-0] [lt=18] submit log success(service_type_=0, last_id=1713774197210958129, limited_id=1713774197210958129) [2024-04-22 16:23:02.723539] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:65) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E8-0-0] [lt=15] submit log with lock success(service_type_=0, last_id=1713774197210958129, limited_id=1713774197210958129) [2024-04-22 16:23:02.723551] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E8-0-0] [lt=9] get number(ret=-4023, service_type_=0, range=1, base_id=1713774182723375243, start_id=0, end_id=94103602987790) [2024-04-22 16:23:02.723571] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E8-0-0] [lt=13][errcode=-4023] get timestamp failed(ret=-4023, ret="OB_EAGAIN") [2024-04-22 16:23:02.723617] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:176) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E8-0-0] [lt=7] handle gts request statistics(total_rt=1161, total_cnt=59, avg_rt=1.935000000000000142e+01, avg_cnt=5.900000000000000355e+00) [2024-04-22 16:23:02.723638] WDIAG [STORAGE.TRANS] process (ob_gts_rpc.cpp:90) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E8-0-0] [lt=20][errcode=-4023] handle request failed(ret=-4023, ret="OB_EAGAIN", arg_={tenant_id:1, srr:[mts=1713774182723607], range_size:1, sender:"172.23.88.131:2882"}) [2024-04-22 16:23:02.727032] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:02.727681] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D98-0-0] [lt=12][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.727738] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D98-0-0] [lt=46][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2931, prev_log_proposal_id:7, committed_end_lsn:{lsn:357365}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.727906] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:721) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=6] update id meta(ret=0, service_type=0, limited_id=1713774197210958129, latest_scn={val:1713774187210958129, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}) [2024-04-22 16:23:02.727957] INFO [STORAGE.TRANS] handle_submit_callback (ob_id_service.cpp:188) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=50] handle submit callback(service_type_=0, success=true, used_ts=4437, log_ts={val:1713774187210958129, v:0}, limited_id=1713774197210958129) [2024-04-22 16:23:02.727970] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:560) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=12] timestamp service handle log callback(ret=0, limited_id=0, log_ts={val:18446744073709551615, v:3}) [2024-04-22 16:23:02.727983] INFO [STORAGE.TRANS] on_success (ob_id_service.cpp:598) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=7] ObPresistIDLogCb on success(ret=0, ret="OB_SUCCESS", *this={log_ts:{val:18446744073709551615, v:3}, log_buf:"", pos:0, id_srv_type:-1, limited_id:0}) [2024-04-22 16:23:02.729320] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:02.729349] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:02.729367] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:02.729375] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:02.739582] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.739607] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={BASE:{msg_type:"Accept Request", id:1, sender:"172.23.88.129:2882", receiver:"172.23.88.130:2882", restart_counter:1, ballot_number:10, debug_ts:{src_construct_ts:"23:02.739529", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1713774182739529}, biggest_min_cluster_version_ever_seen:4.3.0.1}, lease_start_ts_on_proposer:1433560096, lease_interval:4000000, membership_version:{proposal_id:7, config_seq:8}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.739742] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04D-0-0] [lt=10][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:02.739769] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04D-0-0] [lt=26][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:02.739789] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04D-0-0] [lt=19][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:02.739802] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04D-0-0] [lt=12][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:02.740713] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04F-0-0] [lt=114][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:02.740734] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04F-0-0] [lt=20][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:02.740749] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04F-0-0] [lt=14][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:02.740767] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04F-0-0] [lt=17][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:02.740891] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF04F-0-0] [lt=87][errcode=0] compare between invalid priority [2024-04-22 16:23:02.752841] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6B-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.752872] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6B-0-0] [lt=27][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.129:2882"}, cluster_id=1713772869, tenant_id=1, group_id=10, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:02.752886] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6B-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, cluster_id=1713772869, tenant_id=1, arg={addr:"172.23.88.129:2882"}, group_id=10, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:02.752898] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6B-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:02.752909] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:857) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6B-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", addr="172.23.88.130:2882", timeout=2000000) [2024-04-22 16:23:02.753549] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6B-0-0] [lt=7][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=1193, arg={addr:"172.23.88.129:2882"}) [2024-04-22 16:23:02.753580] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:875) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6B-0-0] [lt=25][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:02.753596] INFO [SHARE.LOCATION] merge_leader_from (ob_location_struct.cpp:488) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6B-0-0] [lt=6] [LS_LOCATION] leader will be changed(new_leader={server:"172.23.88.129:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}, old_leader={server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}, exist_replica={server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774181740004, replica_locations:[{server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:6}]}) [2024-04-22 16:23:02.765558] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.765584] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:357237}, curr_lsn:{lsn:357365}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b425, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.766101] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:02.766119] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.766136] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177880657611, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774182766095) [2024-04-22 16:23:02.766149] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774182666068, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:02.766190] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=4884582, log_service_ts=1713774177880657, min_tx_service_ts=4611686018427387, timestamp={val:1713774177880657612, v:0}) [2024-04-22 16:23:02.766171] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7, meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.766207] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=1, last_scn="{val:1713774175784535015, v:0}", last_lsn={lsn:354694}, last_gts={val:1713774177773342916, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:02.766214] INFO [PALF] submit_change_config_meta_req_ (log_engine.h:298) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=42] submit_change_config_meta_req_ success(ret=0, member_list={learner_num:1, learner_array:[{server:"172.23.88.130:2882", timestamp:1, flag:0}]}, msg_proposal_id=7, prev_log_proposal_id=6, prev_lsn={lsn:356459}, prev_mode_pid=7, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:02.768501] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=101, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:02.769764] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D99-0-0] [lt=45][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.769788] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D99-0-0] [lt=20][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2932, prev_log_proposal_id:7, committed_end_lsn:{lsn:357487}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.782635] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ts response task statistics(alloc_count=3582, free_count=1790) [2024-04-22 16:23:02.792522] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.792554] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:02.792567] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:02.792576] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=8][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:02.792584] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=7][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:02.792592] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:02.792599] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:02.792712] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=93][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:02.792764] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=47][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:02.792782] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=17][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:02.792794] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:02.792805] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:02.792813] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=6][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:02.792822] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:02.792831] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:02.792858] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:02.792869] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:02.792881] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=8][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:02.792888] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=6][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:02.792896] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:02.792949] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=49][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:02.792958] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:02.792981] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:02.793001] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:02.793009] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:02.793015] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:02.793033] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7C-0-0] [lt=5][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:02.793044] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:02.793054] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:02.793064] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:02.793088] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=23][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:02.793098] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=8][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1460050, start=1713774182792358, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:02.793110] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:02.793119] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=5][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:02.793219] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=8][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:02.793245] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7C-0-0] [lt=24][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:02.800158] INFO [SQL.ENG] reset_leader_statistics (ob_px_tenant_target_monitor.cpp:382) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=30] reset leader statistics(tenant_id_=1, ret=0, version_=24924322536218, GCTX.server_id_=0) [2024-04-22 16:23:02.800220] INFO [SQL.ENG] refresh_statistics (ob_px_tenant_target_monitor.cpp:125) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=60] refresh global_target_usage_(tenant_id_=1, version_=24924322536218, server_="172.23.88.129:2882", need_refresh_all=false) [2024-04-22 16:23:02.800416] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:02.830663] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10483][LSSysTblUp0][T0][YB42AC175881-000616AB0D0FFAF3-0-0] [lt=26][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774182167448}) [2024-04-22 16:23:02.861709] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:02.866184] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:02.866219] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774177880657612, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774182866178) [2024-04-22 16:23:02.866229] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774182766157, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:02.866257] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:778) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=7, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:02.866275] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:358) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-04-22 16:23:02.866292] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] start TenantWeakReadClusterService(tenant_id=1) [2024-04-22 16:23:02.866999] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=7][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:02.867028] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-04-22 16:23:02.867045] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=15][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-04-22 16:23:02.867053] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=7][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-04-22 16:23:02.867060] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=7][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-04-22 16:23:02.867067] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:02.867072] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=5][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:02.867079] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-04-22 16:23:02.867094] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:02.867104] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:02.867110] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:02.867119] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:02.867124] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:02.867128] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:02.867133] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:02.867145] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=6][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:02.867154] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:02.867160] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:02.867164] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:02.867169] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-04-22 16:23:02.867182] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:02.867191] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:02.867207] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:02.867229] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:02.867237] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:02.867242] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:02.867296] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=7][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:02.867317] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEE-0-0] [lt=20][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-04-22 16:23:02.867326] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-04-22 16:23:02.867332] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:02.867342] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:02.867353] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45cc050, start=1713774182866876, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-04-22 16:23:02.867364] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:02.867369] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:208) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-04-22 16:23:02.867427] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:389) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:02.867457] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:449) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1713774182867456, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=1187, wlock_time=31, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-04-22 16:23:02.867474] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:821) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:02.867482] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=7, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:02.882538] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882596] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=55] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882617] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882633] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882646] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882660] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882711] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882727] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882741] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882760] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882809] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882826] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882839] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882852] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882865] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.882891] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:02.914190] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1713774182914188, bytes: 65563259, bw: 0.013136 MB/s, add_ts: 1006434, add_bytes: 13863 [2024-04-22 16:23:02.929410] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1713774182929409, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007609, add_bytes: 0 [2024-04-22 16:23:02.965682] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.965713] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:357365}, curr_lsn:{lsn:357487}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b49f, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:02.966263] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:02.966330] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187210958130, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774182966258) [2024-04-22 16:23:02.966343] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774182866246, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:02.972731] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9A-0-0] [lt=35][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:02.972762] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9A-0-0] [lt=23][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2933, prev_log_proposal_id:7, committed_end_lsn:{lsn:357609}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.000778] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:03.013032] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=12221, clean_start_pos=3649041, clean_num=125829) [2024-04-22 16:23:03.061814] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:03.065789] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.065819] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:357487}, curr_lsn:{lsn:357609}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b519, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.066368] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:03.066395] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187210958130, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774183066363) [2024-04-22 16:23:03.066408] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774182966354, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:03.070385] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=1028) [2024-04-22 16:23:03.071679] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9B-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.071728] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9B-0-0] [lt=40][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2934, prev_log_proposal_id:7, committed_end_lsn:{lsn:357731}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.071820] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9B-0-0] [lt=34] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=15, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:03.071869] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9B-0-0] [lt=47] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=5910, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:03.071885] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9B-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=3773, avg_log_submit_to_slide_time=5793) [2024-04-22 16:23:03.165350] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=29] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:03.165840] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=59][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.165866] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:357609}, curr_lsn:{lsn:357731}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b593, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.166466] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:03.166606] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=126][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:03.166643] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187210958131, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774183166460) [2024-04-22 16:23:03.166674] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774183066417, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:03.167359] WDIAG [SHARE] get_tenant_primary_zone_score (ob_primary_zone_util.cpp:654) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC73-0-0] [lt=14][errcode=-4016] zone list count unexpected(ret=-4016) [2024-04-22 16:23:03.167385] WDIAG [SHARE] get_tenant_primary_zone_array (ob_primary_zone_util.cpp:613) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC73-0-0] [lt=24][errcode=-4016] failed to get tenant primary zone score(ret=-4016, ret="OB_ERR_UNEXPECTED", tenant_schema={tenant_id:1, schema_version:1, tenant_name:"sys", zone_list:[cnt:0], primary_zone:"", charset_type:2, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:"", zone_replica_attr_array:[cnt:0], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:-1, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2024-04-22 16:23:03.170057] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=7][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.170092] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=25] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:03.170127] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=33][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, ret=-5019) [2024-04-22 16:23:03.170137] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=9][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_service_epoch, db_name=oceanbase) [2024-04-22 16:23:03.170146] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_service_epoch) [2024-04-22 16:23:03.170156] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=7][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.170162] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=5][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.170169] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_service_epoch' doesn't exist [2024-04-22 16:23:03.170174] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:03.170179] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.170186] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:03.170192] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.170202] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:03.170215] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:03.170220] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:03.170234] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.170244] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.170249] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.170254] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.170261] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' , ret=-5019) [2024-04-22 16:23:03.170269] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.170278] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.170292] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=10][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.170305] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.170311] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.170323] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=12][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:03.170336] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=7][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:03.170353] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, aret=-5019, ret=-5019) [2024-04-22 16:23:03.170362] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:03.170373] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:03.170378] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:03.170383] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f3cf3bc0050, start=1713774183169921, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:03.170394] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=10][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:03.170400] WDIAG [SHARE] inner_get_service_epoch_ (ob_service_epoch_proxy.cpp:223) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED30-0-0] [lt=4][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:03.170814] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9C-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.170843] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9C-0-0] [lt=22][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2935, prev_log_proposal_id:7, committed_end_lsn:{lsn:357853}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.170988] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=113] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:03.192464] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [10492][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-04-22 16:23:03.193138] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=14][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.193175] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=30] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:03.193186] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:03.193192] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:03.193199] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:03.193205] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.193210] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.193220] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:23:03.193225] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:03.193229] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.193235] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:03.193239] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.193249] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:03.193264] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=14][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:03.193269] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:03.193299] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=24][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.193305] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.193312] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=6][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.193317] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.193322] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-04-22 16:23:03.193328] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=5][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.193333] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.193343] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=8][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.193355] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.193366] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.193371] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:03.193388] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:03.193394] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10495][RSAsyncTask2][T0][YB42AC175881-000616AB0B0FEBDF-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-04-22 16:23:03.193400] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-04-22 16:23:03.193405] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:03.193410] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:03.193420] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f3ceeb84050, start=1713774183193058, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-04-22 16:23:03.193427] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:03.193584] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-04-22 16:23:03.199516] INFO eloop_handle_sock_event (eloop.c:95) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO sock destroy: sock=0x7f3cd1aac9b8, connection=fd:216:local:172.23.88.129:2882:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:03.199575] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO sk_destroy: s=0x7f3cd1aac9b8 io=0x7f3d00404408 [2024-04-22 16:23:03.199701] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=18][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:03.199729] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=20][errcode=-4122] set config rpc failed(ret=-4122, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:03.199821] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=19] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183199750), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -4122, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:03.199841] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=19][errcode=-4122] execute cmd failed(ret=-4122) [2024-04-22 16:23:03.199851] WDIAG [SQL] open (ob_result_set.cpp:158) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=8][errcode=-4122] execute plan failed(ret=-4122) [2024-04-22 16:23:03.199865] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=10][errcode=-4122] open result set failed(ret=-4122) [2024-04-22 16:23:03.199870] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=5][errcode=-4122] result set open failed(ret=-4122, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:03.199881] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=10][errcode=-4122] execute failed(ret=-4122, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.199889] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=7][errcode=-4122] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4122, err_:"OB_RPC_POST_ERROR", retry_type:0, client_ret:-4122}, need_retry=false) [2024-04-22 16:23:03.199902] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=11][errcode=-4122] result set close failed(ret=-4122) [2024-04-22 16:23:03.199909] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=7][errcode=-4122] result set close failed(ret=-4122) [2024-04-22 16:23:03.199916] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=7][errcode=-4122] failed to close result(close_ret=-4122, ret=-4122) [2024-04-22 16:23:03.199935] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=4][errcode=-4122] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-4122, ret=-4122) [2024-04-22 16:23:03.199943] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10496][RSAsyncTask3][T0][YB42AC175881-000616AB0A1FEC2E-0-0] [lt=7][errcode=-4122] failed to process final(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, aret=-4122, ret=-4122) [2024-04-22 16:23:03.199951] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10496][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4122] execute sql failed(ret=-4122, tenant_id=1, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:03.199974] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10496][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4122] retry_while_no_tenant_resource failed(ret=-4122, tenant_id=1) [2024-04-22 16:23:03.199983] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10496][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4122] execute_write failed(ret=-4122, tenant_id=1, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881', is_user_sql=false) [2024-04-22 16:23:03.199989] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10496][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4122] execute_write failed(ret=-4122, tenant_id=1, sql="ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'") [2024-04-22 16:23:03.199997] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10496][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4122] execute sql failed(ret=-4122, conn=0x7f3cd2206050, start=1713774183198927, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:03.200030] WDIAG [SHARE] store (ob_inner_config_root_addr.cpp:119) [10496][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4122] execute sql failed(ret=-4122, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:03.200043] WDIAG [SHARE] store (ob_rs_mgr.cpp:120) [10496][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] store rs list failed(agent=0, tmp_ret=-4122, addr_list=[{server:"172.23.88.129:2882", role:1, sql_port:2881}, {server:"172.23.88.131:2882", role:2, sql_port:2881}], force=true) [2024-04-22 16:23:03.200072] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10496][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4122] task process failed, start retry(max retry time=9223372036854775806, retry interval=1000000, ret=-4122) [2024-04-22 16:23:03.200542] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.200604] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:03.200615] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:03.200623] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=7][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:03.200633] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=9][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:03.200657] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=22][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.200665] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.200674] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:03.200680] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.200685] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=4][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.200691] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=5][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:03.200696] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=5][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:03.200703] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=6][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:03.200716] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=7][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.200723] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.200729] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=6][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.200734] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.200741] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183199750), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -4122, 'tenant_id', 1, '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:03.200759] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=17][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.200765] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183199750), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -4122, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.200776] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=9][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.200788] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.200794] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC70-0-0] [lt=6][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.206934] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=56][errcode=0] server is initiating(server_id=0, local_seq=14747, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.206965] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221240219, support_ssl=false) [2024-04-22 16:23:03.206996] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221240219, sess.client_addr_="172.23.88.134:41890") [2024-04-22 16:23:03.207021] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f3cf8b72970, session_id:3221240219, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:188}, err:0, last_decode_time:0, last_write_time:1713774183206993, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:03.207282] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10607][sql_nio6][T1][Y0-0000000000000000-0-0] [lt=27] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:03.207345] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=11] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:03.207866] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=22] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:03.207890] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=20] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:03.207911] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=20][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"g(.Lu+s#PKfBVg%}iv7M"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:23:03.207922] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=9][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"g(.Lu+s#PKfBVg%}iv7M"}, ret=-4043) [2024-04-22 16:23:03.207930] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=6][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:23:03.207941] WDIAG [SERVER] process (obmp_connect.cpp:346) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=9][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:23:03.207957] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=4] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:03.208003] INFO [SERVER] free_session (obmp_base.cpp:326) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=7] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221240219, proxy_sessid:0}) [2024-04-22 16:23:03.208027] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=20][errcode=0] server close connection(sessid=3221240219, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:03.208036] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221240219, proxy_sessid=0) [2024-04-22 16:23:03.208045] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=8][errcode=-4016] session is null [2024-04-22 16:23:03.208057] INFO [SERVER] process (obmp_connect.cpp:488) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.23.88.134", client_ip=172.23.88.134, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221240219, proxy_sessid=0, client_sessid=3221240219, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=true, capability=683647754, proxy_capability=1480, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:23:03.208085] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=26][errcode=-4018] get session fail(ret=-4018, sessid=3221240219, proxy_sessid=0) [2024-04-22 16:23:03.208090] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=4][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:03.208115] WDIAG [SERVER] response (obmp_base.cpp:89) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=9][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:03.208121] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD8-0-0] [lt=5][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:03.208145] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f3cf8b72970, session_id:3221240219, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:188}, err:5, last_decode_time:1713774183207238, last_write_time:1713774183208144, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:378, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:03.208177] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221240219, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:03.208192] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f3cf8b72970, session_id:3221240219, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:188}, err:5, last_decode_time:1713774183207238, last_write_time:1713774183208144, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:378, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:03.208203] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221240219, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:03.209625] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10602][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] server is initiating(server_id=0, local_seq=14748, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.209642] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10602][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221240220, support_ssl=false) [2024-04-22 16:23:03.209673] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10602][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221240220, sess.client_addr_="172.23.88.134:41892") [2024-04-22 16:23:03.209687] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10602][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f3ce8bfc030, session_id:3221240220, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:189}, err:0, last_decode_time:0, last_write_time:1713774183209671, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:03.210110] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10602][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:03.210155] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=6] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:03.210650] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=22] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:03.210682] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=29] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:03.210690] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=7][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:":9O_p9g|Z/y7D0_r@&>E"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:23:03.210702] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=11][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:":9O_p9g|Z/y7D0_r@&>E"}, ret=-4043) [2024-04-22 16:23:03.210714] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=11][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:23:03.210722] WDIAG [SERVER] process (obmp_connect.cpp:346) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=7][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:23:03.210737] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=4] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:03.210778] INFO [SERVER] free_session (obmp_base.cpp:326) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=7] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221240220, proxy_sessid:0}) [2024-04-22 16:23:03.210799] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=16][errcode=0] server close connection(sessid=3221240220, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:03.210809] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221240220, proxy_sessid=0) [2024-04-22 16:23:03.210814] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=5][errcode=-4016] session is null [2024-04-22 16:23:03.210819] INFO [SERVER] process (obmp_connect.cpp:488) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=4] MySQL LOGIN(direct_client_ip="172.23.88.134", client_ip=172.23.88.134, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221240220, proxy_sessid=0, client_sessid=3221240220, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=true, capability=683647754, proxy_capability=1480, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:23:03.210856] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=36][errcode=-4018] get session fail(ret=-4018, sessid=3221240220, proxy_sessid=0) [2024-04-22 16:23:03.210864] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=7][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:03.210887] WDIAG [SERVER] response (obmp_base.cpp:89) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=8][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:03.210892] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDB-0-0] [lt=5][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:03.210915] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10602][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f3ce8bfc030, session_id:3221240220, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:189}, err:5, last_decode_time:1713774183210076, last_write_time:1713774183210914, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:03.210940] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10602][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221240220, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:03.210957] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10602][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f3ce8bfc030, session_id:3221240220, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:189}, err:5, last_decode_time:1713774183210076, last_write_time:1713774183210914, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:03.210972] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10602][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221240220, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:03.213341] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:03.224691] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E9-0-0] [lt=11] get number(ret=0, service_type_=0, range=1, base_id=1713774183224687471, start_id=1713774187210958130, end_id=1713774187210958131) [2024-04-22 16:23:03.224717] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10943][T1_L0_G5][T1][YB42AC175883-000616AB095735E9-0-0] [lt=21][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774183224687471, last_request_ts=1713774182782616153, time_delta=442071318, last_gts=1713774187210958129, gts=1713774187210958131, gts_delta=1, compensation_value=44207131) [2024-04-22 16:23:03.233285] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.233310] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={BASE:{msg_type:"Accept Request", id:1, sender:"172.23.88.129:2882", receiver:"172.23.88.130:2882", restart_counter:1, ballot_number:10, debug_ts:{src_construct_ts:"23:03.233247", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1713774183233247}, biggest_min_cluster_version_ever_seen:4.3.0.1}, lease_start_ts_on_proposer:1434053814, lease_interval:4000000, membership_version:{proposal_id:7, config_seq:8}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.233459] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF050-0-0] [lt=17][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:03.233485] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF050-0-0] [lt=24][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:03.233506] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF050-0-0] [lt=20][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:03.233534] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF050-0-0] [lt=27][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:03.234574] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF052-0-0] [lt=117][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:03.234596] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF052-0-0] [lt=21][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:03.234608] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF052-0-0] [lt=12][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:03.234619] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF052-0-0] [lt=10][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:03.234755] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF052-0-0] [lt=87][errcode=0] compare between invalid priority [2024-04-22 16:23:03.240770] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED78-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.240796] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED78-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:03.240808] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED78-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.261923] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:03.265919] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:270) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4122] address in blacklist(ret=-4122, addr="172.23.88.130:2882") [2024-04-22 16:23:03.265939] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.265949] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:357731}, curr_lsn:{lsn:357853}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b60d, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.265977] INFO [LIB] stat (utility.h:1165) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=97, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:03.266011] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1226, avg_group_log_size=122) [2024-04-22 16:23:03.266025] INFO [PALF] submit_log (palf_handle_impl.cpp:453) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, append size=1272) [2024-04-22 16:23:03.266040] INFO [LIB] stat (utility.h:1165) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=141, this=0x7f3cd2d6d1d0, extra_info="id:1") [2024-04-22 16:23:03.266244] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18] ====== tenant freeze timer task ====== [2024-04-22 16:23:03.266494] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.266520] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7, meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.266582] INFO [PALF] submit_change_config_meta_req_ (log_engine.h:298) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=60] submit_change_config_meta_req_ success(ret=0, member_list={learner_num:1, learner_array:[{server:"172.23.88.130:2882", timestamp:1, flag:0}]}, msg_proposal_id=7, prev_log_proposal_id=6, prev_lsn={lsn:356459}, prev_mode_pid=7, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:03.269475] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=67] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3565, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:03.269500] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=1226) [2024-04-22 16:23:03.269553] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:03.271870] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9D-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.271897] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9D-0-0] [lt=23][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2936, prev_log_proposal_id:7, committed_end_lsn:{lsn:357975}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.283182] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:03.304485] INFO eloop_handle_sock_event (eloop.c:95) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO sock destroy: sock=0x7f3d01be7358, connection=fd:211:local:172.23.88.129:2882:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:03.304518] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO sk_destroy: s=0x7f3d01be7358 io=0x7f3d01004408 [2024-04-22 16:23:03.309210] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1482) [11430][T1_L0_G1][T1][YB42AC175883-000616AB092732AF-0-0] [lt=11] receive register_req from acceptor, ignore(self="172.23.88.129:2882", palf_id=1, child={server:"172.23.88.131:2882", region:"default_region", register_time_us:1713774183309416, keepalive_ts:-1}) [2024-04-22 16:23:03.309239] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1501) [11430][T1_L0_G1][T1][YB42AC175883-000616AB092732AF-0-0] [lt=28] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", child={server:"172.23.88.131:2882", region:"default_region", register_time_us:1713774183309416, keepalive_ts:-1}, is_to_leader=true) [2024-04-22 16:23:03.322713] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.366654] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:03.366683] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187210958133, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774183366647) [2024-04-22 16:23:03.366697] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774183166693, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:03.413982] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:03.466058] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.466124] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:357853}, curr_lsn:{lsn:357975}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b687, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.466851] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:778) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=148] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=7, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:03.466969] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:358) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=92] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-04-22 16:23:03.467023] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51] start TenantWeakReadClusterService(tenant_id=1) [2024-04-22 16:23:03.467954] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] Cache replace map node details(ret=0, replace_node_count=0, replace_time=5923, replace_start_pos=754968, replace_num=62914) [2024-04-22 16:23:03.467987] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:03.469551] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEF-0-0] [lt=18][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.469631] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEF-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-04-22 16:23:03.469682] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECEF-0-0] [lt=45][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.470052] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:449) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1713774183470050, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=3105, wlock_time=97, check_leader_time=5, query_version_time=0, persist_version_time=0) [2024-04-22 16:23:03.470126] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=7, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:03.470563] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9E-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.470589] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9E-0-0] [lt=21][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2937, prev_log_proposal_id:7, committed_end_lsn:{lsn:358097}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.566128] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.566152] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:357975}, curr_lsn:{lsn:358097}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b701, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.566957] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:03.567117] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=95][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187210958134, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774183566933) [2024-04-22 16:23:03.567145] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFB85-0-0] [lt=19][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713774183567056}) [2024-04-22 16:23:03.567180] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774183366850, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:03.570354] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9F-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.570383] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173D9F-0-0] [lt=26][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2938, prev_log_proposal_id:7, committed_end_lsn:{lsn:358219}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.598504] INFO eloop_handle_sock_event (eloop.c:95) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO sock destroy: sock=0x7f3cd24d9358, connection=fd:197:local:172.23.88.129:2882:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:03.598543] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO sk_destroy: s=0x7f3cd24d9358 io=0x7f3d00804408 [2024-04-22 16:23:03.604493] INFO eloop_handle_sock_event (eloop.c:95) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO sock destroy: sock=0x7f3d01a05078, connection=fd:192:local:172.23.88.129:2882:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:03.604527] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO sk_destroy: s=0x7f3d01a05078 io=0x7f3cffe04408 [2024-04-22 16:23:03.614377] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=82] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:03.666206] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.666256] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:358097}, curr_lsn:{lsn:358219}, write_buf:{total_size:122, count:1, write_buf:[{buf_:0x7f3cce25b77b, buf_len:122}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.668077] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:03.671349] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED31-0-0] [lt=5][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.671388] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED31-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:03.671402] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED31-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.672392] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DA0-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.672412] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DA0-0-0] [lt=17][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2939, prev_log_proposal_id:7, committed_end_lsn:{lsn:358341}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.698487] INFO eloop_handle_sock_event (eloop.c:95) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO sock destroy: sock=0x7f3cd33ff358, connection=fd:213:local:172.23.88.129:2882:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:03.698525] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO sk_destroy: s=0x7f3cd33ff358 io=0x7f3d01604408 [2024-04-22 16:23:03.699335] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:121, tid:10804}]) [2024-04-22 16:23:03.707796] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.707793] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.707897] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.707913] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.708458] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.708464] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.708544] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.708692] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.708815] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.708883] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.709474] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:469) [10838][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=52] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-04-22 16:23:03.709514] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [10838][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=37] XA scheduler heartbeat task statistics(avg_time=41) [2024-04-22 16:23:03.709902] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=10] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-22 16:23:03.709937] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=23] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=97) [2024-04-22 16:23:03.709966] INFO pn_ratelimit (group.c:589) [10608][IngressService][T0][Y0-0000000000000000-0-0] [lt=31] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:23:03.710339] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC0F-0-0] [lt=16][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-04-22 16:23:03.710356] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC0F-0-0] [lt=16][errcode=-4002] refresh sever list failed(ret=-4002) [2024-04-22 16:23:03.710373] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC0F-0-0] [lt=16][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-04-22 16:23:03.710594] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.712278] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=31][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.712297] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:306) [10844][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=19] ====== traversal_flush timer task ====== [2024-04-22 16:23:03.712306] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.712337] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=42] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:03.712355] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=16][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:23:03.712365] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=9][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:03.712382] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:23:03.712393] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.712403] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.712434] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=23][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:23:03.712471] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.712473] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.712475] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=38][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:03.712484] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.712492] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=8][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:03.712500] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=7][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:23:03.712509] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=8][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:23:03.712519] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=8][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:23:03.712527] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.712536] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:03.712376] INFO [MDS] flush (mds_table_mgr.cpp:172) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC0D-0-0] [lt=15] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:03.712545] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:03.712563] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=16][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:03.712571] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:338) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC0D-0-0] [lt=181] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-22 16:23:03.712584] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.712593] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=6][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.712601] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=6][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.712608] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=6][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.712615] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:23:03.712623] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.712630] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.712648] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=12][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.712664] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.712676] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.712681] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:03.712696] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4E-0-0] [lt=6][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:03.712706] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10899][BlackListServic][T0][YB42AC175881-000616AB07BFEC4E-0-0] [lt=7][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:23:03.712714] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:03.712721] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:03.712727] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:03.712733] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f3cbe430050, start=1713774183711611, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:03.712750] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:03.712759] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:03.712772] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11] start to print blacklist info [2024-04-22 16:23:03.712784] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.712850] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8] ls blacklist refresh finish(cost_time=2882) [2024-04-22 16:23:03.712918] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2024-04-22 16:23:03.713005] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=34] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:03.713050] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=21] cannot_recycle_log_size statistics(cannot_recycle_log_size=358341, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:03.713297] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.713518] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.713716] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.714579] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.715170] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=29] gc stale ls task succ [2024-04-22 16:23:03.715866] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=89] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-22 16:23:03.715914] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=33] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-22 16:23:03.715939] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=20] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=97) [2024-04-22 16:23:03.716516] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=10068, mem_used=76950528) [2024-04-22 16:23:03.717045] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.717315] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=31] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:03.717651] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.717700] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=18] current monitor number(seq_=-1) [2024-04-22 16:23:03.717759] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.718038] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=134] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:03.724406] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=8][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.724462] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=46] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:03.724477] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=14][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:03.724484] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:03.724493] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:03.724501] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.724506] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=5][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.724512] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:03.724517] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.724524] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=6][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.724529] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=5][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:03.724536] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=6][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:03.724541] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:03.724555] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.724561] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.724566] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.724570] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.724580] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:03.724588] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.724593] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.724611] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=11][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.724623] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.724629] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.724636] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:03.724662] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC6A-0-0] [lt=5][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:03.724671] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC6A-0-0] [lt=7][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:03.724678] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:03.724697] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:03.724705] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:03.724710] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:03.724723] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd227a050, start=1713774183723700, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:03.724762] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183723680), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:03.725424] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=15][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.725478] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=48] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:03.725487] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:03.725494] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=7][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:03.725510] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:03.725517] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.725522] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=5][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.725532] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:03.725538] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.725547] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=9][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.725553] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=5][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:03.725557] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=4][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:03.725561] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:03.725568] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.725575] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.725584] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.725589] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.725598] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:03.725603] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=5][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.725610] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.725619] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=7][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.725635] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.725652] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.725656] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:03.725664] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC6B-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:03.725670] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC6B-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:03.725681] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:03.725695] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:03.725700] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:03.725707] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:03.725712] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd45f0050, start=1713774183724817, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:03.725734] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183724805), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:03.725939] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-22 16:23:03.726355] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=9][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.726380] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:03.726393] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:03.726416] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:03.726447] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=31][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:03.726455] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.726459] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.726465] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:03.726471] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.726485] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=13][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.726490] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=4][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:03.726495] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=5][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:03.726499] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:03.726507] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.726513] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.726520] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.726525] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.726529] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:03.726535] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=5][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.726540] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.726559] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=18][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.726569] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.726573] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.726576] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:03.726596] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC6C-0-0] [lt=7][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:03.726605] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC6C-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:03.726610] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:03.726625] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:03.726636] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:03.726634] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=39][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.726645] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:03.726650] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3d1177a050, start=1713774183725796, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:03.726654] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-04-22 16:23:03.726664] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-04-22 16:23:03.726675] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774183725785), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:03.726670] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-04-22 16:23:03.726693] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=22][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-04-22 16:23:03.726699] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.726704] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=5][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.726710] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-04-22 16:23:03.726716] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:03.726722] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=6][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.726732] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:03.726740] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.726747] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=7][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:03.726752] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:03.726756] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:03.726764] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.726780] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=15][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.726786] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.726790] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.726797] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:03.726803] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.726812] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.726821] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=7][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.726832] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.726837] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.726842] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:03.726867] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:03.726886] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0E-0-0] [lt=18][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2024-04-22 16:23:03.726896] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:23:03.726907] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:03.726912] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:03.726919] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1460050, start=1713774183726531, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:23:03.726926] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:03.726943] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:479) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:03.726986] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:141) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2024-04-22 16:23:03.727007] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:252) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] failed to fetch max rule version(ret=-5019) [2024-04-22 16:23:03.727015] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:95) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2024-04-22 16:23:03.727216] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=54] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:03.728600] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=9][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.728633] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:03.728645] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:03.728652] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:03.728658] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:03.728676] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:03.728682] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=5][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:03.728690] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:23:03.728696] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:03.728703] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:03.728710] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:03.728714] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:03.728719] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:03.728729] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:03.728736] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:03.728747] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:03.728771] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=23][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.728786] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=14][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:03.728794] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:03.728799] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-04-22 16:23:03.728804] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=4][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:03.728809] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.728820] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=9][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:03.728830] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.728835] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:03.728839] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC40-0-0] [lt=4][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.729632] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:03.729656] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:03.729665] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:03.729680] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:03.734054] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:131) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] ====== checkpoint timer task ====== [2024-04-22 16:23:03.735732] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1426) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] succ to get rec scn(*this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"L_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:03.735764] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=31] tx ctx memtable get rec scn(this={ObITable:{this:0x7f3cd10be080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f3cd10be080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:03.735789] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:856) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-22 16:23:03.735869] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:03.735937] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:599) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=65] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1}) [2024-04-22 16:23:03.735955] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:470) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2024-04-22 16:23:03.735968] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1713774187210958129, v:0}) [2024-04-22 16:23:03.735974] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-22 16:23:03.735979] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4] get rec log scn(service_type_=2, rec_log_ts={val:1713773717210958135, v:0}) [2024-04-22 16:23:03.736016] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:164) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713773717210958135, v:0}, checkpoint_scn_in_ls_meta={val:1713773717210958135, v:0}, ls_id={id:1}, service_type="DAS_ID") [2024-04-22 16:23:03.736036] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1579) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-22 16:23:03.736047] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:185) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-22 16:23:03.736057] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:192) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-22 16:23:03.737080] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=82][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.737091] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={BASE:{msg_type:"Accept Request", id:1, sender:"172.23.88.129:2882", receiver:"172.23.88.130:2882", restart_counter:1, ballot_number:10, debug_ts:{src_construct_ts:"23:03.737036", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1713774183737036}, biggest_min_cluster_version_ever_seen:4.3.0.1}, lease_start_ts_on_proposer:1434557603, lease_interval:4000000, membership_version:{proposal_id:7, config_seq:8}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.737216] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF053-0-0] [lt=12][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:03.737248] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF053-0-0] [lt=30][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:03.737289] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF053-0-0] [lt=40][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:03.737303] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF053-0-0] [lt=13][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:03.738448] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF055-0-0] [lt=124][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:03.738466] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF055-0-0] [lt=17][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:03.738482] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF055-0-0] [lt=15][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:03.738494] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF055-0-0] [lt=11][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:03.738642] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF055-0-0] [lt=99][errcode=0] compare between invalid priority [2024-04-22 16:23:03.740307] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:03.740327] WDIAG [SERVER] do_renew_lease (ob_lease_state_mgr.cpp:414) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] can't get lease from rs(rs_addr="172.23.88.130:2882", ret=-4122) [2024-04-22 16:23:03.740350] WDIAG [SERVER] renew_lease (ob_lease_state_mgr.cpp:261) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4122] do_renew_lease failed(ret=-4122) [2024-04-22 16:23:03.740367] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:03.740379] INFO [SERVER] renew_lease (ob_lease_state_mgr.cpp:271) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] renew_master_rootserver successfully, try renew lease again [2024-04-22 16:23:03.740417] INFO [SHARE.PT] update_leader_replica_ (ob_inmemory_ls_table.cpp:230) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=6] add leader replica for sys ls(new_replica={modify_time_us:1713774183740415, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info_={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1713774183740415, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774182192732, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:03.740521] INFO [SHARE.PT] update (ob_inmemory_ls_table.cpp:145) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=6] update ls replica(ret=0, ret="OB_SUCCESS", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1713774183740415, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774182192732, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:03.740578] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=56] update ls replica(ret=0, ret="OB_SUCCESS", time_used=172, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2024-04-22 16:23:03.740601] INFO [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:231) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=18] try report sys log stream succeed [2024-04-22 16:23:03.741048] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:03.741079] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=18] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:03.741090] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=11] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:23:03.741109] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:261) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=5] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:03.741123] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:274) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=12][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-04-22 16:23:03.741866] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=10][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774185641139, snapshot={val:1713774187303855825, v:0}) [2024-04-22 16:23:03.741892] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=25][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774185641139) [2024-04-22 16:23:03.741903] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=10][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[19, 20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774185641139, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:3, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f3cd47b6b90, valid:true, source:1, core:{version:{val:1713774187303855825, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774183741825, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5415ab0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5415ad0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5415a70, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5415b20, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:03.742091] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=186][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:03.742100] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=8][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25294, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:0, need_switch_param:0, trans_desc:{this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183741162, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183740077, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cff1ce058, valid:true, source:1, core:{version:{val:1713774187303855825, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774183741825, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:3, 19, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5415ab0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5415ad0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5415a70, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5415b20, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:0], access_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:0], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774185641139, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:03.742315] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=213][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:03.742333] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=15][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:03.742344] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=10][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:03.742349] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=5][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:03.742354] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=5][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:03.742359] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=4][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:03.742364] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=4][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=0) [2024-04-22 16:23:03.742375] WDIAG [SQL.EXE] get_next_row (ob_execute_result.cpp:147) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=10][errcode=-4725] get next row from operator failed(ret=-4725) [2024-04-22 16:23:03.742382] WDIAG [SQL] inner_get_next_row (ob_result_set.cpp:411) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=6][errcode=-4725] get next row from exec result failed(ret=-4725) [2024-04-22 16:23:03.742387] WDIAG [SERVER] open (ob_inner_sql_result.cpp:165) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=4][errcode=-4725] get_next_row failed(ret=-4725, has_tenant_resource()=true) [2024-04-22 16:23:03.742393] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:03.742405] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=11][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:03.742426] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=10] already timeout, do not need sleep(sleep_us=0, remain_us=1998714, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.742484] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=26] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.742497] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=10][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:03.742528] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=16] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183741162, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183740077, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.742579] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=43][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:03.742588] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=8][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:03.742601] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:03.742611] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=4][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:03.742690] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=11][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774185641139, snapshot={val:1713774187303855826, v:0}) [2024-04-22 16:23:03.742708] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=17][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774185641139) [2024-04-22 16:23:03.742716] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=7][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[19, 20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774185641139, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:3, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f3cd47b6b90, valid:true, source:1, core:{version:{val:1713774187303855826, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774183742669, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5415ab0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5415ad0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5415a70, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5415b20, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:03.742863] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=146][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:03.742871] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=7][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25295, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183742233, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183740077, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cff1ce058, valid:true, source:1, core:{version:{val:1713774187303855826, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774183742669, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:3, 19, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5415ab0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5415ad0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5415a70, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5415b20, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:0], access_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:0], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774185641139, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:03.743065] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=193][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:03.743097] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=30][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:03.743114] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=17][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.743130] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=1000, remain_us=1998009, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.744248] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.744282] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183742233, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183740077, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.744451] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=2000, remain_us=1996688, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.746564] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.746594] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183743302, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183740077, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.746732] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=3000, remain_us=1994408, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.749838] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.749862] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183746573, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183740077, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.749996] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=4000, remain_us=1991144, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.752493] INFO eloop_handle_sock_event (eloop.c:95) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO sock destroy: sock=0x7f3cd1aad328, connection=fd:217:local:172.23.88.129:2882:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:03.752532] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO sk_destroy: s=0x7f3cd1aad328 io=0x7f3d00c04408 [2024-04-22 16:23:03.753822] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6C-0-0] [lt=46][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.753849] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6C-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, arg={addr:"172.23.88.129:2882"}, cluster_id=1713772869, tenant_id=1, group_id=10, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:03.753862] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6C-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.23.88.130:2882", timeout=2000000, cluster_id=1713772869, tenant_id=1, arg={addr:"172.23.88.129:2882"}, group_id=10, ret=-4122, ret="OB_RPC_POST_ERROR") [2024-04-22 16:23:03.753870] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6C-0-0] [lt=8][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:03.754119] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.754174] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183749885, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183740077, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.754358] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=5000, remain_us=1986781, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.754387] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:857) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6C-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", addr="172.23.88.130:2882", timeout=2000000) [2024-04-22 16:23:03.754530] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6C-0-0] [lt=125][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=1193, arg={addr:"172.23.88.129:2882"}) [2024-04-22 16:23:03.754607] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:875) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6C-0-0] [lt=66][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:03.754673] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6C-0-0] [lt=48] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, location:{server:"172.23.88.129:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}}]) [2024-04-22 16:23:03.759483] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.759530] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183753150, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183753150, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.759684] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=6000, remain_us=1981456, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.765799] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.765832] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183758673, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.765979] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=7000, remain_us=1975161, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.765983] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1918) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=17] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-22 16:23:03.766004] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1459) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=16] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:03.766015] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1476) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=9] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:03.766036] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2682) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=20] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:03.766875] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.766902] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7, meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.766954] INFO [PALF] submit_change_config_meta_req_ (log_engine.h:298) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=50] submit_change_config_meta_req_ success(ret=0, member_list={learner_num:1, learner_array:[{server:"172.23.88.130:2882", timestamp:1, flag:0}]}, msg_proposal_id=7, prev_log_proposal_id=6, prev_lsn={lsn:356459}, prev_mode_pid=7, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:03.767208] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=104] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713774187255165263, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=-3487965, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1713774183767200, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-22 16:23:03.767346] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=123][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:03.767403] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187255165263, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774183767333) [2024-04-22 16:23:03.767431] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774183567286, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:03.769896] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2401) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=5] Dumping All Cache Objs(alloc_obj_list.count()=6, alloc_obj_list=[{obj_id:99, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:4, added_to_lc:true, mem_used:175160}, {obj_id:100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174200}, {obj_id:101, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173640}, {obj_id:102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173992}, {obj_id:103, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176531}, {obj_id:104, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175513}]) [2024-04-22 16:23:03.769934] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2690) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=35] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:03.773102] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.773141] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183765396, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.773280] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1967860, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.781412] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=26] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.781469] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183773154, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.781622] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=1959517, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.783889] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] gts statistics(tenant_id=1, gts_rpc_cnt=7, get_gts_cache_cnt=188, get_gts_with_stc_cnt=24, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-22 16:23:03.783927] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713774183783825], gts:1713774187303855839, latest_srr:[mts=1713774183783825]}) [2024-04-22 16:23:03.790737] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.790766] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183780911, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.790904] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=10000, remain_us=1950235, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.794278] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7D-0-0] [lt=11][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:03.794361] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7D-0-0] [lt=70] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:03.794378] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7D-0-0] [lt=15][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.801014] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.801053] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183789759, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.801197] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=11000, remain_us=1939942, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.812324] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.812366] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183800800, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.812488] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] get number(ret=0, service_type_=0, range=1, base_id=1713774183812486948, start_id=1713774187303855844, end_id=1713774187303855845) [2024-04-22 16:23:03.812508] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=19][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774183812486948, last_request_ts=1713774183711593006, time_delta=100893942, last_gts=1713774187255165264, gts=1713774187303855845, gts_delta=48690580, compensation_value=10089394) [2024-04-22 16:23:03.812551] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=12000, remain_us=1928589, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.814671] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:03.824681] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.824722] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183811649, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.824876] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=1916264, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.837005] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFB91-0-0] [lt=54][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774183740597}) [2024-04-22 16:23:03.837994] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.838033] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183824483, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.838165] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=14000, remain_us=1902974, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.840982] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:480) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=47] socket need_disconn(n=0, errno=11) [2024-04-22 16:23:03.841049] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:508) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=63] server connection closed, fd: 186, addr: "172.23.88.130:55266" [2024-04-22 16:23:03.844201] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] sock regist: 0x7f3cfd8726b0 fd=192 [2024-04-22 16:23:03.844230] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] accept new connection, fd:192, src_addr:172.23.88.130:55288 [2024-04-22 16:23:03.844254] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x7f3cfd872780 fd=194 [2024-04-22 16:23:03.844263] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] accept new connection, fd:194, src_addr:172.23.88.130:55290 [2024-04-22 16:23:03.844398] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:192, src_addr:172.23.88.130:55288 [2024-04-22 16:23:03.844415] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:192, gid:0x100000002 [2024-04-22 16:23:03.844463] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x7f3cfd8726b0 fd=186 [2024-04-22 16:23:03.844462] INFO pkts_sk_init (pkts_sk_factory.h:23) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO set pkts_sk_t sock_id s=0x7f3cd3372048, s->id=589823 [2024-04-22 16:23:03.844473] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] accept new connection, fd:186, src_addr:172.23.88.130:55292 [2024-04-22 16:23:03.844475] INFO pkts_sk_new (pkts_sk_factory.h:51) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO sk_new: s=0x7f3cd3372048 [2024-04-22 16:23:03.844494] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:194, src_addr:172.23.88.130:55290 [2024-04-22 16:23:03.844500] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] PNIO dispatch fd to certain group, fd:194, gid:0x100000005 [2024-04-22 16:23:03.844522] INFO eloop_regist (eloop.c:47) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sock regist: 0x7f3cd3372048 fd=192 [2024-04-22 16:23:03.844547] INFO on_accept (listenfd.c:39) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO accept new connection, ns=0x7f3cd3372048, fd=fd:192:local:172.23.88.129:2882:remote:172.23.88.130:55288 [2024-04-22 16:23:03.844568] WDIAG listenfd_handle_event (listenfd.c:71) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:03.844617] INFO pkts_sk_init (pkts_sk_factory.h:23) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO set pkts_sk_t sock_id s=0x7f3d01bf89b8, s->id=524285 [2024-04-22 16:23:03.844680] INFO pkts_sk_new (pkts_sk_factory.h:51) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=61] PNIO sk_new: s=0x7f3d01bf89b8 [2024-04-22 16:23:03.844780] INFO eloop_regist (eloop.c:47) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO sock regist: 0x7f3d01bf89b8 fd=194 [2024-04-22 16:23:03.844844] INFO on_accept (listenfd.c:39) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO accept new connection, ns=0x7f3d01bf89b8, fd=fd:194:local:172.23.88.129:2882:remote:172.23.88.130:55290 [2024-04-22 16:23:03.844941] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] auth mothod is NONE, the fd will be dispatched, fd:186, src_addr:172.23.88.130:55292 [2024-04-22 16:23:03.844954] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] PNIO dispatch fd to certain group, fd:186, gid:0x100000001 [2024-04-22 16:23:03.845005] INFO pkts_sk_init (pkts_sk_factory.h:23) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO set pkts_sk_t sock_id s=0x7f3d01bf9328, s->id=524286 [2024-04-22 16:23:03.845035] INFO pkts_sk_new (pkts_sk_factory.h:51) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO sk_new: s=0x7f3d01bf9328 [2024-04-22 16:23:03.845068] INFO eloop_regist (eloop.c:47) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO sock regist: 0x7f3d01bf9328 fd=186 [2024-04-22 16:23:03.845087] INFO on_accept (listenfd.c:39) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO accept new connection, ns=0x7f3d01bf9328, fd=fd:186:local:172.23.88.129:2882:remote:172.23.88.130:55292 [2024-04-22 16:23:03.845084] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10943][T1_L0_G5][T1][YB42AC175881-000616AB0B5FEDC5-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774183838133], gts:1713774187313945242, latest_srr:[mts=1713774183783825]}) [2024-04-22 16:23:03.845100] INFO [STORAGE.TRANS] handle_gts_err_response (ob_gts_source.cpp:704) [10943][T1_L0_G5][T1][YB42AC175881-000616AB0B5FEDC5-0-0] [lt=15] handle gts err response(ret=0, ret="OB_SUCCESS", err_msg={tenant_id:1, srr:[mts=1713774180780206], status:-4038, sender:"172.23.88.130:2882"}, *this={tenant_id:1, gts_local_cache:{srr:[mts=1713774183838133], gts:1713774187313945242, latest_srr:[mts=1713774183783825]}, server:"172.23.88.129:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-04-22 16:23:03.845107] INFO [STORAGE.TRANS] handle_gts_err_response (ob_gts_source.cpp:704) [10944][T1_L0_G5][T1][YB42AC175881-000616AB0B5FEDC3-0-0] [lt=6] handle gts err response(ret=0, ret="OB_SUCCESS", err_msg={tenant_id:1, srr:[mts=1713774179779160], status:-4038, sender:"172.23.88.130:2882"}, *this={tenant_id:1, gts_local_cache:{srr:[mts=1713774183838133], gts:1713774187313945242, latest_srr:[mts=1713774183783825]}, server:"172.23.88.129:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-04-22 16:23:03.845104] WDIAG listenfd_handle_event (listenfd.c:71) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=78][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:03.845152] INFO [STORAGE.TRANS] handle_gts_err_response (ob_gts_source.cpp:704) [10943][T1_L0_G5][T1][YB42AC175881-000616AB0B5FEDC2-0-0] [lt=21] handle gts err response(ret=0, ret="OB_SUCCESS", err_msg={tenant_id:1, srr:[mts=1713774179278627], status:-4038, sender:"172.23.88.130:2882"}, *this={tenant_id:1, gts_local_cache:{srr:[mts=1713774183838133], gts:1713774187313945242, latest_srr:[mts=1713774183783825]}, server:"172.23.88.129:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-04-22 16:23:03.845175] WDIAG listenfd_handle_event (listenfd.c:71) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:03.845163] WDIAG [PALF] receive_log_ (palf_handle_impl.cpp:3047) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C0E-0-0] [lt=14][errcode=-4109] can not receive log(ret=-4109, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", msg_proposal_id=6, lsn={lsn:356581}, local proposal_id=7, role=1, state=2, is_sync_enabled=true) [2024-04-22 16:23:03.845187] INFO [STORAGE.TRANS] handle_gts_err_response (ob_gts_source.cpp:704) [10945][T1_L0_G5][T1][YB42AC175881-000616AB0B5FEDC0-0-0] [lt=6] handle gts err response(ret=0, ret="OB_SUCCESS", err_msg={tenant_id:1, srr:[mts=1713774178276848], status:-4038, sender:"172.23.88.130:2882"}, *this={tenant_id:1, gts_local_cache:{srr:[mts=1713774183838133], gts:1713774187313945242, latest_srr:[mts=1713774183783825]}, server:"172.23.88.129:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-04-22 16:23:03.846039] WDIAG [PALF] can_receive_mode_meta (log_mode_mgr.cpp:695) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F13-0-0] [lt=44][errcode=-4016] can_receive_mode_meta failed(palf_id=1, self="172.23.88.129:2882", proposal_id=6, mode_meta={version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:03.846066] WDIAG [PALF] receive_mode_meta (palf_handle_impl.cpp:2779) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F13-0-0] [lt=26][errcode=0] can_receive_mode_meta failed(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, proposal_id=6, mode_meta={version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:03.846078] INFO [PALF] receive_mode_meta (palf_handle_impl.cpp:2793) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F13-0-0] [lt=11] receive_mode_meta finish(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", proposal_id=6, is_applied_mode_meta=true, mode_meta={version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:03.846094] INFO [PALF] handle_request (log_request_handler.cpp:335) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F13-0-0] [lt=16] receive_mode_meta success(ret=0, palf_id=1, server="172.23.88.130:2882", req={msg_proposal_id:6, meta:{version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:true}, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-04-22 16:23:03.846390] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x7f3cfd8726b0 fd=197 [2024-04-22 16:23:03.846409] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] accept new connection, fd:197, src_addr:172.23.88.130:55302 [2024-04-22 16:23:03.846483] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:197, src_addr:172.23.88.130:55302 [2024-04-22 16:23:03.846497] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] PNIO dispatch fd to certain group, fd:197, gid:0x100000003 [2024-04-22 16:23:03.846553] INFO pkts_sk_init (pkts_sk_factory.h:23) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO set pkts_sk_t sock_id s=0x7f3d01bfa048, s->id=262142 [2024-04-22 16:23:03.846569] INFO pkts_sk_new (pkts_sk_factory.h:51) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO sk_new: s=0x7f3d01bfa048 [2024-04-22 16:23:03.846597] INFO eloop_regist (eloop.c:47) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO sock regist: 0x7f3d01bfa048 fd=197 [2024-04-22 16:23:03.846616] INFO on_accept (listenfd.c:39) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO accept new connection, ns=0x7f3d01bfa048, fd=fd:197:local:172.23.88.129:2882:remote:172.23.88.130:55302 [2024-04-22 16:23:03.846668] WDIAG listenfd_handle_event (listenfd.c:71) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:03.846685] INFO [STORAGE.TRANS] handle_gts_err_response (ob_gts_source.cpp:704) [10943][T1_L0_G5][T1][YB42AC175881-000616AB0B5FEDC4-0-0] [lt=25] handle gts err response(ret=0, ret="OB_SUCCESS", err_msg={tenant_id:1, srr:[mts=1713774180279673], status:-4038, sender:"172.23.88.130:2882"}, *this={tenant_id:1, gts_local_cache:{srr:[mts=1713774183838133], gts:1713774187313945242, latest_srr:[mts=1713774183783825]}, server:"172.23.88.129:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-04-22 16:23:03.846787] INFO [STORAGE.TRANS] handle_gts_err_response (ob_gts_source.cpp:704) [10943][T1_L0_G5][T1][YB42AC175881-000616AB0B5FEDC1-0-0] [lt=22] handle gts err response(ret=0, ret="OB_SUCCESS", err_msg={tenant_id:1, srr:[mts=1713774178778062], status:-4038, sender:"172.23.88.130:2882"}, *this={tenant_id:1, gts_local_cache:{srr:[mts=1713774183838133], gts:1713774187313945242, latest_srr:[mts=1713774183783825]}, server:"172.23.88.129:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-04-22 16:23:03.852301] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=22] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.852342] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183837400, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183754257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.852525] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=15000, remain_us=1888614, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.853339] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=88][errcode=0] server is initiating(server_id=0, local_seq=14749, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.853358] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221240221, support_ssl=false) [2024-04-22 16:23:03.853392] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221240221, sess.client_addr_="172.23.88.134:41922") [2024-04-22 16:23:03.853410] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f3cf8b6c030, session_id:3221240221, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:195}, err:0, last_decode_time:0, last_write_time:1713774183853388, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:03.853691] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10603][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:03.853758] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=10] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:03.854214] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=20] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:03.854233] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=15] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:03.854241] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=6][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"Q~J4d|eb=G0XMUGBUe}d"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:23:03.854251] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"Q~J4d|eb=G0XMUGBUe}d"}, ret=-4043) [2024-04-22 16:23:03.854262] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=9][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:23:03.854278] WDIAG [SERVER] process (obmp_connect.cpp:346) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=14][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:23:03.854299] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=6] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:03.854372] INFO [SERVER] free_session (obmp_base.cpp:326) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=23] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221240221, proxy_sessid:0}) [2024-04-22 16:23:03.854397] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=19][errcode=0] server close connection(sessid=3221240221, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:03.854410] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221240221, proxy_sessid=0) [2024-04-22 16:23:03.854421] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=10][errcode=-4016] session is null [2024-04-22 16:23:03.854454] INFO [SERVER] process (obmp_connect.cpp:488) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=31] MySQL LOGIN(direct_client_ip="172.23.88.134", client_ip=172.23.88.134, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221240221, proxy_sessid=0, client_sessid=3221240221, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=true, capability=683647754, proxy_capability=1480, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:23:03.854482] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=25][errcode=-4018] get session fail(ret=-4018, sessid=3221240221, proxy_sessid=0) [2024-04-22 16:23:03.854491] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=8][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:03.854507] WDIAG [SERVER] response (obmp_base.cpp:89) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=5][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:03.854521] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDD-0-0] [lt=12][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:03.854540] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f3cf8b6c030, session_id:3221240221, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:195}, err:5, last_decode_time:1713774183853662, last_write_time:1713774183854538, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:377, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:03.854565] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221240221, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:03.854575] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f3cf8b6c030, session_id:3221240221, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:195}, err:5, last_decode_time:1713774183853662, last_write_time:1713774183854538, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:377, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:03.854601] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] connection close(sessid=3221240221, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:03.856330] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] server is initiating(server_id=0, local_seq=14750, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.856346] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221240222, support_ssl=false) [2024-04-22 16:23:03.856368] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221240222, sess.client_addr_="172.23.88.134:41924") [2024-04-22 16:23:03.856377] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f3cf8b6cd70, session_id:3221240222, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:198}, err:0, last_decode_time:0, last_write_time:1713774183856366, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:03.856803] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10607][sql_nio6][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:03.856838] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=4] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:03.857190] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=7] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:03.857208] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=14] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:03.857215] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=6][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"\ht.(|Y(Zn:uadRNqmP'"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:23:03.857224] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"\ht.(|Y(Zn:uadRNqmP'"}, ret=-4043) [2024-04-22 16:23:03.857236] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=10][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:23:03.857243] WDIAG [SERVER] process (obmp_connect.cpp:346) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=6][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:23:03.857260] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=5] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:03.857308] INFO [SERVER] free_session (obmp_base.cpp:326) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=14] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221240222, proxy_sessid:0}) [2024-04-22 16:23:03.857329] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=15][errcode=0] server close connection(sessid=3221240222, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:03.857337] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221240222, proxy_sessid=0) [2024-04-22 16:23:03.857341] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=4][errcode=-4016] session is null [2024-04-22 16:23:03.857346] INFO [SERVER] process (obmp_connect.cpp:488) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=4] MySQL LOGIN(direct_client_ip="172.23.88.134", client_ip=172.23.88.134, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221240222, proxy_sessid=0, client_sessid=3221240222, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=true, capability=683647754, proxy_capability=1480, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:23:03.857370] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=22][errcode=-4018] get session fail(ret=-4018, sessid=3221240222, proxy_sessid=0) [2024-04-22 16:23:03.857379] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=8][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:03.857397] WDIAG [SERVER] response (obmp_base.cpp:89) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=7][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:03.857419] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBD9-0-0] [lt=21][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:03.857422] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f3cf8b6cd70, session_id:3221240222, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:198}, err:5, last_decode_time:1713774183856779, last_write_time:1713774183857421, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:03.857454] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221240222, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:03.857469] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f3cf8b6cd70, session_id:3221240222, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:198}, err:5, last_decode_time:1713774183856779, last_write_time:1713774183857421, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:03.857483] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221240222, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:03.866335] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] tx gc loop thread is running(MTL_ID()=1) [2024-04-22 16:23:03.866357] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] try gc retain ctx [2024-04-22 16:23:03.866385] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:748) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] get_max_applied_scn(scn={val:1713774187255165264, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:7, palf_committed_end_lsn_:{lsn:358341}, palf_committed_end_scn_:{val:1713774187255165264, v:0}, last_check_scn_:{val:1713774187255165264, v:0}, max_applied_cb_scn_:{val:1713774187255165264, v:0}}) [2024-04-22 16:23:03.866461] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1023) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=67] get_min_unreplayed_log_info(lsn={lsn:356581}, scn={val:1713774177880657608, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774182065832, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:356581}, next_to_submit_scn_:{val:1713774177880657608, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:356459}, end_lsn:{lsn:356581}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774177880657607, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774177880657607, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:1991013568, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774177880657607, v:0}}}}}) [2024-04-22 16:23:03.866922] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=38] LogLoopThread round_cost_time(us)(round_cost_time=162) [2024-04-22 16:23:03.867304] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=94] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1713774187255165264, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1713774183867269}, version_delta=-3387862) [2024-04-22 16:23:03.867512] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:253) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=53] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:03.867619] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=71][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.867639] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={push_log_type:"PUSH_LOG", msg_proposal_id:7, prev_log_proposal_id:7, prev_lsn:{lsn:358219}, curr_lsn:{lsn:358341}, write_buf:{total_size:113, count:1, write_buf:[{buf_:0x7f3cce25b7f5, buf_len:113}]}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.867646] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=26] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.867681] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183851390, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183846042, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.867854] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=16000, remain_us=1873285, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.868191] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:03.870183] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=37, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:03.872601] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DA1-0-0] [lt=25][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.872639] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DA1-0-0] [lt=32][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, prev_log_id:2940, prev_log_proposal_id:7, committed_end_lsn:{lsn:358454}}, server="172.23.88.130:2882") [2024-04-22 16:23:03.883918] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.883990] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884029] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884042] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.884079] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884100] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884121] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884138] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884151] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884165] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884186] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884112] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183867329, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183846042, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.884207] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884248] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884288] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884308] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884327] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884348] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:03.884551] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=17000, remain_us=1856589, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.899173] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=30][errcode=0] server is initiating(server_id=0, local_seq=14751, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.901636] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=61][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.901799] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=57] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.901809] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1535af0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240223, assoc_session_id:3221240223, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183900482, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183898262, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240223, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd1535af0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.901889] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183883744, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183872880, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.902002] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.902280] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=18000, remain_us=1838860, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.903605] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=59][errcode=0] server is initiating(server_id=0, local_seq=14752, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.906273] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.906325] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.906375] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=27] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1537a10, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240224, assoc_session_id:3221240224, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183906049, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183902912, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240224, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1537a10}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.906878] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.909216] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=54][errcode=0] server is initiating(server_id=0, local_seq=14753, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.910871] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.910902] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=30][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.910929] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=20] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1539930, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240225, assoc_session_id:3221240225, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183910666, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183908281, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240225, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1539930}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.911101] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.914807] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=30][errcode=0] server is initiating(server_id=0, local_seq=14754, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.914764] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1713774183914761, bytes: 65591410, bw: 0.026832 MB/s, add_ts: 1000573, add_bytes: 28151 [2024-04-22 16:23:03.915823] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=30] get number(ret=0, service_type_=0, range=1, base_id=1713774183915820896, start_id=1713774187313945254, end_id=1713774187313945255) [2024-04-22 16:23:03.915861] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=35][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774183915820896, last_request_ts=1713774183812486948, time_delta=103333948, last_gts=1713774187303855845, gts=1713774187313945255, gts_delta=10089409, compensation_value=10333394) [2024-04-22 16:23:03.915919] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.915937] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=17][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.915962] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=17] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd153b850, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240226, assoc_session_id:3221240226, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183915230, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183914004, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240226, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153b850}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.916129] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.920518] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.920642] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183901603, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183898262, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.921020] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=1820120, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.921136] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=27][errcode=0] server is initiating(server_id=0, local_seq=14755, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.922382] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.922421] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=37][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.922495] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=63] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd153d770, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240227, assoc_session_id:3221240227, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183922029, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183920911, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240227, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153d770}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.922699] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.928914] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=34][errcode=0] server is initiating(server_id=0, local_seq=14756, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.930076] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.930111] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=33][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.930142] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=23] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd153f690, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240228, assoc_session_id:3221240228, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183929988, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183927733, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240228, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153f690}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.930358] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.937117] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1713774183937113, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007704, add_bytes: 0 [2024-04-22 16:23:03.937727] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=33][errcode=0] server is initiating(server_id=0, local_seq=14757, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.939119] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.939159] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=38][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.939188] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=18] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15415b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240229, assoc_session_id:3221240229, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183939004, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183937894, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240229, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15415b0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.939467] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.940284] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=63] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.940384] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183919796, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183914004, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.941065] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1800076, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.947500] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=38][errcode=0] server is initiating(server_id=0, local_seq=14758, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.949989] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.950054] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=61][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.950103] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=33] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15434d0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240230, assoc_session_id:3221240230, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183949278, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183947054, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240230, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15434d0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.950567] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.953841] INFO [RPC.OBRPC] check_connect (ob_net_keepalive.cpp:548) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=20] connect ok, fd: 204, conn: "172.23.88.130:2882" [2024-04-22 16:23:03.959201] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=64][errcode=0] server is initiating(server_id=0, local_seq=14759, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.960203] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.960232] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=28][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.960252] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=13] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15453f0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240231, assoc_session_id:3221240231, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183959305, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183958177, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240231, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15453f0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.960428] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.961304] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=49] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.961404] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183940107, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183939003, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.961870] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=21000, remain_us=1779270, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.967075] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.23.88.130:2882") [2024-04-22 16:23:03.967137] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={msg_proposal_id:7, meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:true}, server="172.23.88.130:2882") [2024-04-22 16:23:03.967399] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:03.967507] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=104][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187255165264, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774183967386) [2024-04-22 16:23:03.967538] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774183767526, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:03.970584] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=47][errcode=0] server is initiating(server_id=0, local_seq=14760, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.971121] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.971140] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=18][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.971151] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=7] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1547310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240232, assoc_session_id:3221240232, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183970695, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183969543, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240232, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1547310}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.971253] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.982657] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=16][errcode=0] server is initiating(server_id=0, local_seq=14761, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.983118] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=45] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:03.983217] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183961522, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183958177, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:03.983638] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=22000, remain_us=1757502, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:03.983999] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.984043] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=42][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.984076] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=22] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1549230, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240233, assoc_session_id:3221240233, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183983013, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183981898, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240233, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1549230}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.984319] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:03.996327] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10955][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=39][errcode=0] server is initiating(server_id=0, local_seq=14762, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:03.998475] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10955][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=96][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:03.998569] WDIAG push_back (ob_fixed_array.h:71) [10955][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:03.998591] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10955][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=21][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:03.998617] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10955][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=14] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cc567d930, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240234, assoc_session_id:3221240234, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183997596, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183995377, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc8a7048, id:3221240234, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cc567d930}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:03.998954] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10955][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.005963] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=44] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.006182] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774183983013, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183981898, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.006592] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=23000, remain_us=1734548, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.012232] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=35][errcode=0] server is initiating(server_id=0, local_seq=14763, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.014064] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.014116] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.014154] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=24] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154b150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240235, assoc_session_id:3221240235, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184013266, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184011107, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240235, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154b150}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.014537] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.015311] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20734345216, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:04.028926] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=63][errcode=0] server is initiating(server_id=0, local_seq=14764, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.029870] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=44] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.029969] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184005533, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774183995377, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.030235] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] get number(ret=0, service_type_=0, range=1, base_id=1713774184030233241, start_id=1713774187324278668, end_id=1713774187324278669) [2024-04-22 16:23:04.030283] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=44][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774184030233241, last_request_ts=1713774183915820896, time_delta=114412345, last_gts=1713774187313945255, gts=1713774187324278669, gts_delta=10333413, compensation_value=11441234) [2024-04-22 16:23:04.030412] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1710728, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.030880] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.030933] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=50][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.030972] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154d070, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240236, assoc_session_id:3221240236, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184029859, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184028745, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240236, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154d070}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.031297] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.042258] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x7f3cfd8726b0 fd=188 [2024-04-22 16:23:04.042330] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=52] [ussl] accept new connection, fd:188, src_addr:172.23.88.130:55308 [2024-04-22 16:23:04.046210] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=62][errcode=0] server is initiating(server_id=0, local_seq=14765, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.048227] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.048284] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=54][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.048322] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154ef90, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240237, assoc_session_id:3221240237, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184047878, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184045633, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240237, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154ef90}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.048716] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.054737] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=85] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.054844] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184029859, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184028745, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.055301] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] will sleep(sleep_us=25000, remain_us=1685840, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.064706] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=54][errcode=0] server is initiating(server_id=0, local_seq=14766, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.066706] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.066765] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=56][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.066808] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=28] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1550eb0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240238, assoc_session_id:3221240238, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184066294, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184063914, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240238, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1550eb0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.067199] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.067511] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:778) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=7, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:04.067593] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:358) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=68] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-04-22 16:23:04.067615] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] start TenantWeakReadClusterService(tenant_id=1) [2024-04-22 16:23:04.068329] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:04.069478] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECF0-0-0] [lt=11][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.069551] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECF0-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-04-22 16:23:04.069859] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:449) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1713774184069857, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=2277, wlock_time=47, check_leader_time=4, query_version_time=0, persist_version_time=0) [2024-04-22 16:23:04.069937] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=7, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:04.071499] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:04.080590] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=99] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.080709] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184054740, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184045633, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.081049] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=26000, remain_us=1660091, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.084228] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10892][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=56][errcode=0] server is initiating(server_id=0, local_seq=14767, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.086182] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10892][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=70][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.086276] WDIAG push_back (ob_fixed_array.h:71) [10892][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.086305] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10892][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=27][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.086335] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10892][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=17] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15a6970, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240239, assoc_session_id:3221240239, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184085863, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184083513, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc89e1f8, id:3221240239, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15a6970}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.086722] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10892][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.104380] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=48][errcode=0] server is initiating(server_id=0, local_seq=14768, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.106409] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.106492] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=80][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.106531] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1552dd0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240240, assoc_session_id:3221240240, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184106165, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184103918, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240240, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1552dd0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.106844] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.107356] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=45] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.107469] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184080005, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184063914, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.107813] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=27000, remain_us=1633328, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.125740] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=48][errcode=0] server is initiating(server_id=0, local_seq=14769, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.127744] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.127807] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=60][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.127844] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=24] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1554cf0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240241, assoc_session_id:3221240241, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184127634, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184125316, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240241, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1554cf0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.128133] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.135015] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=41] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.135116] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184107271, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184103918, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.135406] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] get number(ret=0, service_type_=0, range=1, base_id=1713774184135403082, start_id=1713774187335719916, end_id=1713774187335719917) [2024-04-22 16:23:04.135490] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=80][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774184135403082, last_request_ts=1713774184030233241, time_delta=105169841, last_gts=1713774187324278669, gts=1713774187335719917, gts_delta=11441247, compensation_value=10516984) [2024-04-22 16:23:04.135678] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=28000, remain_us=1605462, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.148075] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=52][errcode=0] server is initiating(server_id=0, local_seq=14770, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.150014] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.150063] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=46][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.150101] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=24] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1558b30, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240242, assoc_session_id:3221240242, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184149152, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184146786, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240242, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1558b30}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.150502] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.153837] INFO [RPC.OBRPC] try_read_response (ob_net_keepalive.cpp:609) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=38] whitewash, addr: "172.23.88.130:2882", last_write_ts_=1713774181352753, last_read_ts_=1713774184153821 [2024-04-22 16:23:04.163876] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=40] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.164002] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184134447, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184125316, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.164518] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=29000, remain_us=1576623, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.166251] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED36-0-0] [lt=45] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:04.166332] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=31] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:04.167609] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:04.167652] WDIAG [SHARE] get_tenant_primary_zone_score (ob_primary_zone_util.cpp:654) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC74-0-0] [lt=15][errcode=-4016] zone list count unexpected(ret=-4016) [2024-04-22 16:23:04.167665] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:04.167697] WDIAG [SHARE] get_tenant_primary_zone_array (ob_primary_zone_util.cpp:613) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC74-0-0] [lt=40][errcode=-4016] failed to get tenant primary zone score(ret=-4016, ret="OB_ERR_UNEXPECTED", tenant_schema={tenant_id:1, schema_version:1, tenant_name:"sys", zone_list:[cnt:0], primary_zone:"", charset_type:2, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:"", zone_replica_attr_array:[cnt:0], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:-1, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2024-04-22 16:23:04.167735] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187313945245, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774184167597) [2024-04-22 16:23:04.167787] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774183967559, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:04.171386] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=50][errcode=0] server is initiating(server_id=0, local_seq=14771, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.173321] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.173375] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=51][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.173509] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED32-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.173414] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1556c10, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240243, assoc_session_id:3221240243, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184172883, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184172049, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240243, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1556c10}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.173579] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED32-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:04.173831] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.193736] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [10492][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=20] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-04-22 16:23:04.193782] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=58] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.193893] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184164032, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184146786, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.194335] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=30000, remain_us=1546806, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.195944] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=53][errcode=0] server is initiating(server_id=0, local_seq=14772, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.195989] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7C-0-0] [lt=38][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.196043] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7C-0-0] [lt=37] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:04.196072] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10494][RSAsyncTask1][T1][YB42AC175881-000616AB0A0FEC7C-0-0] [lt=27][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.197957] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.198007] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=48][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.198047] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd155aa50, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240244, assoc_session_id:3221240244, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184197480, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184195231, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240244, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155aa50}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.198375] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.203762] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=77][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1997547) [2024-04-22 16:23:04.203846] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=74][errcode=-8001] [2024-04-22 16:23:04.203905] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.204693] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1997547) [2024-04-22 16:23:04.204744] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=46][errcode=-8001] [2024-04-22 16:23:04.204793] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.205472] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=24][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1997547) [2024-04-22 16:23:04.205508] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=32][errcode=-8001] [2024-04-22 16:23:04.205540] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.206388] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1997547) [2024-04-22 16:23:04.206423] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=32][errcode=-8001] [2024-04-22 16:23:04.206475] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=41][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:04.206638] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=73] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184206553), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.206675] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=32][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:04.206701] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:04.206719] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=14][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:04.206735] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:04.206758] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.206805] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=29] already timeout, do not need sleep(sleep_us=0, remain_us=1993568, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.206827] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:04.206848] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.206858] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=9][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.206866] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:04.206901] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=20][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:04.208172] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1993027) [2024-04-22 16:23:04.208227] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=49][errcode=-8001] [2024-04-22 16:23:04.208277] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.208778] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC71-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.208838] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC71-0-0] [lt=39] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.208876] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC71-0-0] [lt=34][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.209035] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1993027) [2024-04-22 16:23:04.209073] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=35][errcode=-8001] [2024-04-22 16:23:04.209105] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.210142] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1993027) [2024-04-22 16:23:04.210211] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=64][errcode=-8001] [2024-04-22 16:23:04.210257] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.211000] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1993027) [2024-04-22 16:23:04.211035] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=31][errcode=-8001] [2024-04-22 16:23:04.211057] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:04.211163] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184211086), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.211198] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=31][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:04.211217] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:04.211228] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=9][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:04.211238] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:04.211259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.211287] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15] will sleep(sleep_us=1000, remain_us=1989086, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.212417] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:04.212493] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=69][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.212518] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.212541] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=20][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:04.212570] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:04.213417] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC72-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.213589] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC72-0-0] [lt=141] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.213982] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1987235) [2024-04-22 16:23:04.214031] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=45][errcode=-8001] [2024-04-22 16:23:04.214082] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.214871] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=44][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1987235) [2024-04-22 16:23:04.214910] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=35][errcode=-8001] [2024-04-22 16:23:04.214943] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.215642] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1987235) [2024-04-22 16:23:04.215687] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=42][errcode=-8001] [2024-04-22 16:23:04.215715] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.215965] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=73] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:04.216287] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1987235) [2024-04-22 16:23:04.216317] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=27][errcode=-8001] [2024-04-22 16:23:04.216336] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:04.216520] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=33] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184216373), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.216551] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=27][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:04.216567] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=14][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:04.216576] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:04.216585] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=7][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:04.216597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.216620] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11] will sleep(sleep_us=2000, remain_us=1983753, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.218703] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC73-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.218736] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:04.218770] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=28][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.218798] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.218811] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:04.218840] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:04.218798] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC73-0-0] [lt=68] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.220178] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1980982) [2024-04-22 16:23:04.220229] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=46][errcode=-8001] [2024-04-22 16:23:04.220273] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.221067] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10960][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=67][errcode=0] server is initiating(server_id=0, local_seq=14773, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.221091] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1980982) [2024-04-22 16:23:04.221121] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=27][errcode=-8001] [2024-04-22 16:23:04.221154] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.221964] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1980982) [2024-04-22 16:23:04.222025] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=58][errcode=-8001] [2024-04-22 16:23:04.222059] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.222777] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1980982) [2024-04-22 16:23:04.222819] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=38][errcode=-8001] [2024-04-22 16:23:04.222843] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:04.222932] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184222868), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.222927] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10960][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=57][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.222962] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:04.222978] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=14][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:04.222991] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:04.223007] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=14][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:04.223009] WDIAG push_back (ob_fixed_array.h:71) [10960][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.223019] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=3, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.223031] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10960][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=21][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.223043] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13] will sleep(sleep_us=3000, remain_us=1977330, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.223058] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10960][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=15] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd155c970, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240245, assoc_session_id:3221240245, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184222701, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184222700, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240245, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155c970}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.223395] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10960][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.223673] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [10825][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=24] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-22 16:23:04.224350] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC74-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.224422] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC74-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.224610] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=80] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.224705] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184193026, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184177331, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.225142] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=31000, remain_us=1515998, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.226163] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:3, local_retry_times:3, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:04.226241] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=71][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.226253] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.226260] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=7][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:04.226277] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:04.227291] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1973647) [2024-04-22 16:23:04.227329] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=34][errcode=-8001] [2024-04-22 16:23:04.227361] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.228123] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1973647) [2024-04-22 16:23:04.228162] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=35][errcode=-8001] [2024-04-22 16:23:04.228205] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=24] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.228912] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1973647) [2024-04-22 16:23:04.228944] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=29][errcode=-8001] [2024-04-22 16:23:04.228971] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.230099] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1973647) [2024-04-22 16:23:04.230136] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=33][errcode=-8001] [2024-04-22 16:23:04.230158] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:04.230252] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=23] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184230184), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.230278] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:04.230296] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:04.230305] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:04.230315] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:04.230327] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=4, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.230351] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13] will sleep(sleep_us=4000, remain_us=1970021, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.231991] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC75-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.232062] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC75-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.232697] INFO pktc_sk_new (pktc_sk_factory.h:78) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO sk_new: s=0x7f3cd33d4048 [2024-04-22 16:23:04.232924] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF056-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:04.232968] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF056-0-0] [lt=41][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.233017] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF056-0-0] [lt=46][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.233055] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF056-0-0] [lt=36][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:04.233235] INFO pktc_do_connect (pktc_post.h:19) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=63] PNIO sk_new: sk=0x7f3cd33d4048, fd=189 [2024-04-22 16:23:04.233323] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=51] [ussl] write client fd succ, fd:189, gid:0x100000004, need_send_negotiation:1 [2024-04-22 16:23:04.233348] INFO eloop_regist (eloop.c:47) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO sock regist: 0x7f3cd33d4048 fd=189 [2024-04-22 16:23:04.233395] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO sock not ready: 0x7f3cd33d4048, fd=189 [2024-04-22 16:23:04.233406] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] sock regist: 0x7f3cfd8074d0 fd=189 [2024-04-22 16:23:04.233649] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] client send negotiation message succ, fd:189, addr:172.23.88.129:52410, auth_method:NONE, gid:0x100000004 [2024-04-22 16:23:04.233781] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] give back fd to origin epoll succ, client_fd:189, client_epfd:131, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:04.233845] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sock connect OK: 0x7f3cd33d4048 fd:189:local:172.23.88.129:52410:remote:172.23.88.130:2882 [2024-04-22 16:23:04.234477] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:4, local_retry_times:4, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:04.234523] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=39][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.234541] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.234550] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:04.234586] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:04.234749] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF058-0-0] [lt=217][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:04.234859] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF058-0-0] [lt=106][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.234905] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF058-0-0] [lt=43][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.234960] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF058-0-0] [lt=52][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:04.235537] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF058-0-0] [lt=389][errcode=0] compare between invalid priority [2024-04-22 16:23:04.235874] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1965261) [2024-04-22 16:23:04.235913] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=35][errcode=-8001] [2024-04-22 16:23:04.235951] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.236497] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF057-0-0] [lt=38][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:04.236546] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF057-0-0] [lt=45][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.236581] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF057-0-0] [lt=33][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.236604] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1965261) [2024-04-22 16:23:04.236622] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] [2024-04-22 16:23:04.236644] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=7] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.236610] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF057-0-0] [lt=27][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:04.237485] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1965261) [2024-04-22 16:23:04.237569] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=76][errcode=-8001] [2024-04-22 16:23:04.237628] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.238337] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=44][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1965261) [2024-04-22 16:23:04.238409] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=66][errcode=-8001] [2024-04-22 16:23:04.238475] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:04.238742] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=52] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184238532), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.238788] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=40][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:04.238811] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=20][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:04.238825] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:04.238838] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:04.238855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=5, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.238890] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19] will sleep(sleep_us=5000, remain_us=1961484, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.240624] INFO acceptfd_handle_first_readable_event (handle-event.c:382) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=33] [ussl] recv non-negotiation message, the fd will be dispatched, fd:188, src_addr:172.23.88.130:55308, magic:0x78563412 [2024-04-22 16:23:04.240695] INFO dispatch_accept_fd_to_certain_group (group.c:637) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=56] PNIO dispatch fd to oblistener, fd:188 [2024-04-22 16:23:04.240737] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC76-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.240851] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC76-0-0] [lt=98] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.240858] INFO [RPC] read_client_magic (ob_listener.cpp:219) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=116] read negotiation msg(rcv_byte=20) [2024-04-22 16:23:04.240898] INFO [RPC] read_client_magic (ob_listener.cpp:239) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] read_client_magic, (client_magic=7386785325300370467, index=0) [2024-04-22 16:23:04.240934] INFO [RPC] trace_connection_info (ob_listener.cpp:285) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] oblistener receive connection from(src_addr=172.23.88.130, src_port=55308) [2024-04-22 16:23:04.241000] INFO [RPC] do_one_event (ob_listener.cpp:407) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=64] dispatch to(client_magic=7386785325300370467, index=0) [2024-04-22 16:23:04.241029] INFO [RPC] connection_redispatch (ob_listener.cpp:261) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] dipatch(conn_fd=188, count=1, index=0, wrfd=97) [2024-04-22 16:23:04.241101] INFO [RPC] connection_redispatch (ob_listener.cpp:267) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] dispatch success!(conn_fd=188, wrfd=97) [2024-04-22 16:23:04.241240] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:456) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=7] new connection established, fd: 188, addr: "172.23.88.130:55308" [2024-04-22 16:23:04.242944] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED79-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.243021] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED79-0-0] [lt=57] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:04.243115] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED79-0-0] [lt=88][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.244134] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:5, local_retry_times:5, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:04.244212] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=67][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.244240] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=25][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:04.244260] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:04.244291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:04.246636] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=27][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1954390) [2024-04-22 16:23:04.246692] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=51][errcode=-8001] [2024-04-22 16:23:04.246740] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.247268] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=108][errcode=0] server is initiating(server_id=0, local_seq=14774, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.247473] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=46][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1954390) [2024-04-22 16:23:04.247557] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=78][errcode=-8001] [2024-04-22 16:23:04.247615] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.248305] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=31][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1954390) [2024-04-22 16:23:04.248352] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=43][errcode=-8001] [2024-04-22 16:23:04.248392] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.249040] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=23][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1954390) [2024-04-22 16:23:04.249106] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=60][errcode=-8001] [2024-04-22 16:23:04.249144] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:04.249191] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.249216] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.249305] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=46] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184249194), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.249244] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=17] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd155e890, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240246, assoc_session_id:3221240246, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184248774, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184248773, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240246, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155e890}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.249351] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=40][errcode=-8001] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.249401] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=6000, remain_us=1950972, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.249612] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.251413] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC77-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.251529] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC77-0-0] [lt=97] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.256530] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.256611] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184223926, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184222700, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.256848] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.257092] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=32000, remain_us=1484048, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.257694] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.258506] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.259574] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184259406), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.259657] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=7000, remain_us=1940716, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.261630] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC78-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.261705] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC78-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.267569] INFO pktc_sk_new (pktc_sk_factory.h:78) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO sk_new: s=0x7f3cd33fe048 [2024-04-22 16:23:04.267531] INFO [PALF] submit_change_config_meta_req_ (log_engine.h:298) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=41] submit_change_config_meta_req_ success(ret=0, member_list={learner_num:1, learner_array:[{server:"172.23.88.130:2882", timestamp:1, flag:0}]}, msg_proposal_id=7, prev_log_proposal_id=6, prev_lsn={lsn:356459}, prev_mode_pid=7, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:04.267925] INFO pktc_do_connect (pktc_post.h:19) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=86] PNIO sk_new: sk=0x7f3cd33fe048, fd=201 [2024-04-22 16:23:04.267990] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=38] [ussl] write client fd succ, fd:201, gid:0x100000006, need_send_negotiation:1 [2024-04-22 16:23:04.268039] INFO eloop_regist (eloop.c:47) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=46] PNIO sock regist: 0x7f3cd33fe048 fd=201 [2024-04-22 16:23:04.268079] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=28] PNIO sock not ready: 0x7f3cd33fe048, fd=201 [2024-04-22 16:23:04.268087] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] sock regist: 0x7f3cfd872040 fd=201 [2024-04-22 16:23:04.268382] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] client send negotiation message succ, fd:201, addr:172.23.88.129:52412, auth_method:NONE, gid:0x100000006 [2024-04-22 16:23:04.268481] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] give back fd to origin epoll succ, client_fd:201, client_epfd:145, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:04.268533] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=57] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:04.268600] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sock connect OK: 0x7f3cd33fe048 fd:201:local:172.23.88.129:52412:remote:172.23.88.130:2882 [2024-04-22 16:23:04.268667] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.269213] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.269861] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.270619] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184270530), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.270691] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1929682, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.272614] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC79-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.272688] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC79-0-0] [lt=55] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.272781] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] sock regist: 0x7f3cfd8726b0 fd=208 [2024-04-22 16:23:04.272869] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=74] [ussl] accept new connection, fd:208, src_addr:172.23.88.130:55310 [2024-04-22 16:23:04.272969] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] [ussl] auth mothod is NONE, the fd will be dispatched, fd:208, src_addr:172.23.88.130:55310 [2024-04-22 16:23:04.273009] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] PNIO dispatch fd to certain group, fd:208, gid:0x100000004 [2024-04-22 16:23:04.273104] INFO pkts_sk_init (pkts_sk_factory.h:23) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO set pkts_sk_t sock_id s=0x7f3cd3205358, s->id=589821 [2024-04-22 16:23:04.273156] INFO pkts_sk_new (pkts_sk_factory.h:51) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=50] PNIO sk_new: s=0x7f3cd3205358 [2024-04-22 16:23:04.273234] INFO eloop_regist (eloop.c:47) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO sock regist: 0x7f3cd3205358 fd=208 [2024-04-22 16:23:04.273283] INFO on_accept (listenfd.c:39) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO accept new connection, ns=0x7f3cd3205358, fd=fd:208:local:172.23.88.129:2882:remote:172.23.88.130:55310 [2024-04-22 16:23:04.273394] WDIAG listenfd_handle_event (listenfd.c:71) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:04.273527] INFO [PALF] get_log (palf_handle_impl.cpp:3994) [11430][T1_L0_G1][T1][YB42AC175881-000616AB08DFEC54-0-0] [lt=17] submit fetch log task success(ret=0, palf_id=1, server="172.23.88.130:2882", fetch_type=2, prev_lsn={lsn:356459}, start_lsn={lsn:356581}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=6) [2024-04-22 16:23:04.273568] INFO [PALF] handle (fetch_log_engine.cpp:284) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=11] handle fetch_log_task(fetch_log_task={timestamp_us:1713774184273500, id:1, server:"172.23.88.130:2882", fetch_type:"FETCH_MODE_META", proposal_id:7, prev_lsn:{lsn:356459}, start_lsn:{lsn:356581}, log_size:41943040, log_count:2048, accepted_mode_pid:6}) [2024-04-22 16:23:04.273670] INFO [PALF] read (log_cache.cpp:100) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=63] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=366, hit_cnt=3, read_cnt=3, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:04.273860] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3768) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=55][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x7f3ccc7f7000, next_round_pread_size:2121728, curr_read_pos:1873, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1873, log_storage_:{IteratorStorage:{start_lsn:{lsn:356581}, end_lsn:{lsn:358454}, read_buf:{buf_len_:8192, buf_:0x7f3ccc7f7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187313945245, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:57, proposal_id:7, committed_lsn:{lsn:358341}, max_scn:{val:1713774187313945245, v:0}, accumulated_checksum:3095296725, log_id:2940, flag:0}}, init_mode_version:0, accumulate_checksum:3095296725, curr_entry_is_padding:0, padding_entry_size:57, padding_entry_scn:{val:1713774187313945245, v:0}}}, replayable_point={val:18446744073709551615, v:3}, prev_lsn={lsn:358341}) [2024-04-22 16:23:04.274168] INFO pktc_sk_new (pktc_sk_factory.h:78) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sk_new: s=0x7f3d01ba4048 [2024-04-22 16:23:04.274094] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3705) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=116] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, prev_lsn={lsn:356459}, iterator={iterator_impl:{buf_:0x7f3ccc7f7000, next_round_pread_size:2121728, curr_read_pos:1873, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1873, log_storage_:{IteratorStorage:{start_lsn:{lsn:356581}, end_lsn:{lsn:358454}, read_buf:{buf_len_:8192, buf_:0x7f3ccc7f7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187313945245, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:57, proposal_id:7, committed_lsn:{lsn:358341}, max_scn:{val:1713774187313945245, v:0}, accumulated_checksum:3095296725, log_id:2940, flag:0}}, init_mode_version:0, accumulate_checksum:3095296725, curr_entry_is_padding:0, padding_entry_size:57, padding_entry_scn:{val:1713774187313945245, v:0}}}, fetch_log_count=2048, has_fetched_log_count=15) [2024-04-22 16:23:04.274290] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=129] submit_change_mode_meta_req success(ret=0, palf_id=1, server="172.23.88.130:2882", mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=6) [2024-04-22 16:23:04.274353] INFO [PALF] fetch_log_from_storage (palf_handle_impl.cpp:3490) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT FETCH LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=83886080) [2024-04-22 16:23:04.274396] INFO [LIB] stat (utility.h:1165) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT FETCH LOG TASK IN QUEUE TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=44, this=0x7f3ce8ff4600, extra_info="") [2024-04-22 16:23:04.274422] INFO [LIB] stat (utility.h:1165) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FETCH LOG EXECUTE COST TIME](cur_stat_count=2, stat_interval=1000000, avg (count/cost)=483, this=0x7f3ce8ff4830, extra_info="") [2024-04-22 16:23:04.274495] INFO pktc_do_connect (pktc_post.h:19) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO sk_new: sk=0x7f3d01ba4048, fd=211 [2024-04-22 16:23:04.274492] INFO [PALF] handle (fetch_log_engine.cpp:321) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=68] handle fetch log task(ret=0, palf_id=1, handle_cost_time_us=832, fetch_log_task={timestamp_us:1713774184273500, id:1, server:"172.23.88.130:2882", fetch_type:"FETCH_MODE_META", proposal_id:7, prev_lsn:{lsn:356459}, start_lsn:{lsn:356581}, log_size:41943040, log_count:2048, accepted_mode_pid:6}, fetch_stat={total_size:1873, group_log_cnt:15, read_cost:0, get_cost:0, send_cost:109}) [2024-04-22 16:23:04.274546] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] write client fd succ, fd:211, gid:0x100000003, need_send_negotiation:1 [2024-04-22 16:23:04.274561] INFO eloop_regist (eloop.c:47) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO sock regist: 0x7f3d01ba4048 fd=211 [2024-04-22 16:23:04.274551] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=65][errcode=0] server is initiating(server_id=0, local_seq=14775, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.274595] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] sock regist: 0x7f3cfd8074d0 fd=211 [2024-04-22 16:23:04.274584] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO sock not ready: 0x7f3d01ba4048, fd=211 [2024-04-22 16:23:04.274755] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] client send negotiation message succ, fd:211, addr:172.23.88.129:52414, auth_method:NONE, gid:0x100000003 [2024-04-22 16:23:04.274812] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] give back fd to origin epoll succ, client_fd:211, client_epfd:124, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:04.274899] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO sock connect OK: 0x7f3d01ba4048 fd:211:local:172.23.88.129:52414:remote:172.23.88.130:2882 [2024-04-22 16:23:04.276544] WDIAG [PALF] handle_prepare_response (log_mode_mgr.cpp:660) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08DC33AA-0-0] [lt=68][errcode=-4109] handle_prepare_response not in PREPARE state(ret=-4109, palf_id=1, self="172.23.88.129:2882", new_proposal_id=9223372036854775807, server="172.23.88.130:2882", msg_proposal_id=7, mode_meta={version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state="MODE_INIT") [2024-04-22 16:23:04.276645] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.276673] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=27][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.276702] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=17] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15607b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240247, assoc_session_id:3221240247, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184276533, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184275425, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240247, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15607b0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.276639] WDIAG [PALF] handle_prepare_response (palf_handle_impl.cpp:2747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08DC33AA-0-0] [lt=93][errcode=-4109] log_mode_mgr.handle_prepare_response failed(ret=-4109, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, proposal_id=7, accept_proposal_id=6, last_lsn={lsn:356703}, log_mode_meta={version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:04.276953] WDIAG [PALF] handle_request (log_request_handler.cpp:259) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08DC33AA-0-0] [lt=307][errcode=-4109] PalfHandleImpl handle_prepare_response failed(ret=-4109, server="172.23.88.130:2882", palf_id=1, req={msg_proposal_id:7, vote_granted:true, log_proposal_id:6, max_flushed_lsn:{lsn:356703}, log_mode_meta:{version:1, proposal_id:6, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, committed_end_lsn:{lsn:356581}}, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-04-22 16:23:04.277025] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.280044] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.280924] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.281701] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.282508] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184282352), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.282590] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=1917784, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.284494] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7A-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.284566] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7A-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.284997] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:04.289331] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=51] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.289467] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184256503, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184255403, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.289971] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=33000, remain_us=1451169, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.292674] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.293429] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.294231] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.294974] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184294870), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.295055] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=10000, remain_us=1905319, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.296843] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7B-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.296913] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7B-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.303107] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=79][errcode=0] server is initiating(server_id=0, local_seq=14776, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.305095] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.305148] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=51][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.305187] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15242d0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240248, assoc_session_id:3221240248, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184304735, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184302470, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240248, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15242d0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.305584] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.306283] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.307049] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.307723] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.308503] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184308365), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.308581] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=11000, remain_us=1891793, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.310606] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7C-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.310678] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7C-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.310924] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=18][errcode=-4018] try free send task failed(ret=-4018) [2024-04-22 16:23:04.320771] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.321379] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.322035] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.322528] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.322979] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184322800), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.323056] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=12000, remain_us=1877317, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.323240] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=45] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.323342] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184288831, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184288837, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.323816] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1417325, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.324923] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7D-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.324993] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7D-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.332332] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=53][errcode=0] server is initiating(server_id=0, local_seq=14777, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.334269] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.334320] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=48][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.334359] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15261f0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240249, assoc_session_id:3221240249, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184333651, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184331427, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240249, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15261f0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.334716] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.336546] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.337228] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.337954] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.338944] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184338773), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.339027] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=13000, remain_us=1861346, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.340870] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7E-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.340939] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7E-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.353094] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.354124] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.354811] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.355841] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184355743), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.355923] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1844451, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.357736] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7F-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.357844] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC7F-0-0] [lt=89] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.358098] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=52] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.358208] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184322512, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184322511, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.358633] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1382508, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.362847] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] server is initiating(server_id=0, local_seq=14778, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.364662] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.364716] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=52][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.364750] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=21] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1528110, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240250, assoc_session_id:3221240250, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184363826, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184361497, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240250, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1528110}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.365125] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.367848] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:04.367928] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187313945245, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774184367833) [2024-04-22 16:23:04.367976] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774184167811, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:04.371986] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.372720] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.373415] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.374330] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184374146), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.374432] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=1825942, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.376301] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC80-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.376375] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC80-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.390837] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.391548] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.392274] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.393112] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184393015), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.393191] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=16000, remain_us=1807182, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.393901] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=45] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.394099] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184358121, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184354701, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.394353] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10951][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=3][errcode=0] server is initiating(server_id=0, local_seq=14779, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.394574] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=36000, remain_us=1346566, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.395164] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC81-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.395236] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC81-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.396265] WDIAG push_back (ob_fixed_array.h:71) [10951][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.396318] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10951][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=51][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.396358] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10951][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=26] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cc5630eb0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240251, assoc_session_id:3221240251, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184395330, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184393057, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240251, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cc5630eb0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.396805] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10951][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.411198] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.412232] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.413001] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.413982] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184413899), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.414080] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=17000, remain_us=1786294, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.415863] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC82-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.415939] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC82-0-0] [lt=58] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.416692] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=71] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20732248064, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:04.426871] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=59][errcode=0] server is initiating(server_id=0, local_seq=14780, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.429372] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.429431] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=57][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.429526] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=79] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd152a030, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240252, assoc_session_id:3221240252, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184428807, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184426601, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240252, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd152a030}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.429925] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.430828] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.430976] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184394189, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184393057, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.431358] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=37000, remain_us=1309783, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.432159] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.432836] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.433567] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.434236] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184434164), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.434308] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=18000, remain_us=1766065, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.436202] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC83-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.436283] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC83-0-0] [lt=57] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.453612] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.454350] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.455040] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.455802] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184455707), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.455875] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=1744498, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.458161] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC84-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.458262] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC84-0-0] [lt=75] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.460831] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] server is initiating(server_id=0, local_seq=14781, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.462992] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.463036] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=42][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.463070] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=20] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd152bf50, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240253, assoc_session_id:3221240253, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184462350, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184460131, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240253, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd152bf50}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.463506] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.468593] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=43] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.468759] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:04.468765] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=37] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184431050, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184426601, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.469257] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=38000, remain_us=1271884, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.476429] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.477143] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.477993] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.478809] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184478736), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.478880] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1721493, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.480940] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC85-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.481019] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC85-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.495193] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=67][errcode=0] server is initiating(server_id=0, local_seq=14782, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.497085] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.497184] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=97][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.497224] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd152de70, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240254, assoc_session_id:3221240254, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184496199, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184494005, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240254, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd152de70}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.497609] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.500028] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.500710] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.501314] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.501981] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184501891), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.502054] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=21000, remain_us=1698320, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.503826] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC86-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.503902] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC86-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.507506] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=48] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.507604] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184468078, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184460131, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.508003] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=39000, remain_us=1233137, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.524515] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.525206] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.525922] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.526726] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184526605), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.526808] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=22000, remain_us=1673565, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.528668] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC87-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.528738] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC87-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.530306] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] server is initiating(server_id=0, local_seq=14783, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.532487] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.532551] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=60][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.532593] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=26] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd152fd90, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240255, assoc_session_id:3221240255, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184531828, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184530985, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240255, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd152fd90}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.532960] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.547212] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=47] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.547309] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184507344, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184501771, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.547743] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=40000, remain_us=1193397, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.550158] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.550797] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.551410] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.552088] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184552020), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.552161] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=23000, remain_us=1648213, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.554027] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC88-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.554100] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC88-0-0] [lt=55] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.566703] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=54][errcode=0] server is initiating(server_id=0, local_seq=14784, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.568054] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:04.568119] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187313945245, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774184568041) [2024-04-22 16:23:04.568150] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774184368010, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:04.568990] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.569047] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=54][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.569099] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=33] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1531cb0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240256, assoc_session_id:3221240256, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184568591, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184566393, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240256, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1531cb0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.569524] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.569583] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFBA9-0-0] [lt=22][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774184569487}) [2024-04-22 16:23:04.576499] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.577287] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.577987] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.578624] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184578556), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.578698] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=24000, remain_us=1621676, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.580638] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC89-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.580743] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC89-0-0] [lt=87] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.587961] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=67] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.588053] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184546346, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184533203, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.588482] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=41000, remain_us=1152659, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.603508] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.603755] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=51][errcode=0] server is initiating(server_id=0, local_seq=14785, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.604152] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.604840] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.604969] INFO keepalive_check (time_wheel.c:76) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd33d4048,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:04.605546] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184605476), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.605613] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=25000, remain_us=1594760, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.605825] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.605875] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=47][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.605917] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=26] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1533bd0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240257, assoc_session_id:3221240257, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184605363, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184605362, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240257, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1533bd0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.606279] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.607400] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8A-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.607510] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8A-0-0] [lt=88] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.617352] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20732248064, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:04.629785] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=50] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.629910] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=3] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184587519, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184577517, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.630324] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=42000, remain_us=1110816, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.631656] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.632330] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.632828] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.633335] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184633214), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.633403] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=26000, remain_us=1566971, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.635167] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8B-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.635236] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8B-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.641516] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=92][errcode=0] server is initiating(server_id=0, local_seq=14786, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.643520] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.643582] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=60][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.643624] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=26] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1535af0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240258, assoc_session_id:3221240258, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184642815, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184643462, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240258, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1535af0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.644002] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.660492] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.661020] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.661615] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.662251] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184662160), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.662316] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=27000, remain_us=1538057, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.664067] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8C-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.664134] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8C-0-0] [lt=50] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.668162] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:778) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=7, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:04.668228] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:358) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-04-22 16:23:04.668263] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] start TenantWeakReadClusterService(tenant_id=1) [2024-04-22 16:23:04.668928] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:04.669970] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECF1-0-0] [lt=28][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.670033] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECF1-0-0] [lt=46] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-04-22 16:23:04.670076] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECF1-0-0] [lt=40][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.670408] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:449) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1713774184670406, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=2191, wlock_time=78, check_leader_time=4, query_version_time=0, persist_version_time=0) [2024-04-22 16:23:04.670529] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=7, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:04.672619] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=47] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.672723] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184629856, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184605362, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.673168] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=3] get number(ret=0, service_type_=0, range=1, base_id=1713774184673165554, start_id=1713774187389445976, end_id=1713774187389445977) [2024-04-22 16:23:04.673260] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=88][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774184673165554, last_request_ts=1713774184567493367, time_delta=105672187, last_gts=1713774187377511770, gts=1713774187389445977, gts_delta=11934206, compensation_value=10567218) [2024-04-22 16:23:04.673368] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=43000, remain_us=1067772, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.675994] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED33-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.676060] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED33-0-0] [lt=50] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:04.676106] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED33-0-0] [lt=41][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.680072] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] server is initiating(server_id=0, local_seq=14787, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.682021] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.682072] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.682121] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=35] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1537a10, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240259, assoc_session_id:3221240259, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184681308, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184679082, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240259, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1537a10}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.682421] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.690356] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.690809] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.691355] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.692036] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184691964), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.692104] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=28000, remain_us=1508269, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.693836] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8D-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.693915] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8D-0-0] [lt=58] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.706987] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.707499] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.707888] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.707891] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.708344] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.708370] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.708453] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.708455] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.708572] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.708684] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.709912] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.711942] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.712061] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.712099] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:222) [10510][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=60] [ASYNC TASK QUEUE](queue_size=0) [2024-04-22 16:23:04.712545] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.712609] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=34] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.713086] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.713566] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.713617] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.714667] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.715273] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=28] gc stale ls task succ [2024-04-22 16:23:04.716545] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=32] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.716697] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10233, mem_used=79030272) [2024-04-22 16:23:04.716644] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184672395, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184661294, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.717035] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.717098] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=44000, remain_us=1024043, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.717569] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=50] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:04.717851] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.718135] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.718196] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:04.719396] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=89][errcode=0] server is initiating(server_id=0, local_seq=14788, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.721045] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.721255] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.721316] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=58][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.721357] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=28] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1539930, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240260, assoc_session_id:3221240260, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184720142, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184720142, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240260, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1539930}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.721613] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.721685] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.722135] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.722692] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184722656), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.722722] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=29000, remain_us=1477651, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.724626] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8E-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.724711] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8E-0-0] [lt=63] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.725995] INFO [SQL.ENG] reset_leader_statistics (ob_px_tenant_target_monitor.cpp:382) [10963][T1_L0_G0][T1][YB42AC175883-000616AB09A735CA-0-0] [lt=2] reset leader statistics(tenant_id_=1, ret=0, version_=24924324462057, GCTX.server_id_=0) [2024-04-22 16:23:04.726021] INFO [SQL.ENG] process (ob_px_rpc_processor.cpp:613) [10963][T1_L0_G0][T1][YB42AC175883-000616AB09A735CA-0-0] [lt=24] need refresh all(tenant_id=1, arg_.need_refresh_all_=true, follower_version=24924294718515, prev_leader_server_id=0, leader_server_id=0) [2024-04-22 16:23:04.727394] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:04.730112] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:04.730169] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:04.730190] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:04.730207] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:04.739215] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF059-0-0] [lt=202][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:04.739282] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF059-0-0] [lt=63][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.739351] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF059-0-0] [lt=66][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.739396] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF059-0-0] [lt=43][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:04.740369] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05A-0-0] [lt=114][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:04.740430] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05A-0-0] [lt=57][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.740535] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05A-0-0] [lt=100][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.740628] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05A-0-0] [lt=90][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:04.740992] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05B-0-0] [lt=331][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:04.741062] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05B-0-0] [lt=65][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.741161] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05B-0-0] [lt=93][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:04.741247] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05B-0-0] [lt=82][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:04.741736] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05B-0-0] [lt=351][errcode=0] compare between invalid priority [2024-04-22 16:23:04.752394] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.752897] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.753370] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.753941] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184753913), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.753969] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=30000, remain_us=1446404, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.754585] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8F-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.754611] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC8F-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.755144] INFO pktc_sk_new (pktc_sk_factory.h:78) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO sk_new: s=0x7f3cd24d9358 [2024-04-22 16:23:04.755396] INFO pktc_do_connect (pktc_post.h:19) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=43] PNIO sk_new: sk=0x7f3cd24d9358, fd=213 [2024-04-22 16:23:04.755516] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=88] [ussl] write client fd succ, fd:213, gid:0x100000001, need_send_negotiation:1 [2024-04-22 16:23:04.755560] INFO eloop_regist (eloop.c:47) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO sock regist: 0x7f3cd24d9358 fd=213 [2024-04-22 16:23:04.755599] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO sock not ready: 0x7f3cd24d9358, fd=213 [2024-04-22 16:23:04.755641] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] sock regist: 0x7f3cfd872040 fd=213 [2024-04-22 16:23:04.755806] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=46] [ussl] client send negotiation message succ, fd:213, addr:172.23.88.129:52416, auth_method:NONE, gid:0x100000001 [2024-04-22 16:23:04.755927] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=53] [ussl] give back fd to origin epoll succ, client_fd:213, client_epfd:110, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:04.755966] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO sock connect OK: 0x7f3cd24d9358 fd:213:local:172.23.88.129:52416:remote:172.23.88.130:2882 [2024-04-22 16:23:04.759611] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=33][errcode=0] server is initiating(server_id=0, local_seq=14789, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.761258] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.761313] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=52][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.761383] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=121] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.761357] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=31] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd153b850, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240261, assoc_session_id:3221240261, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184760052, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184758954, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240261, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153b850}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.761484] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=89][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.761535] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=30] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184715694, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184691342, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.761677] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.761955] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=45000, remain_us=979185, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.768058] INFO pktc_sk_new (pktc_sk_factory.h:78) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO sk_new: s=0x7f3cd33ff358 [2024-04-22 16:23:04.768022] INFO [PALF] submit_change_config_meta_req_ (log_engine.h:298) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=122] submit_change_config_meta_req_ success(ret=0, member_list={learner_num:1, learner_array:[{server:"172.23.88.130:2882", timestamp:1, flag:0}]}, msg_proposal_id=7, prev_log_proposal_id=6, prev_lsn={lsn:356459}, prev_mode_pid=7, config_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:04.768273] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:04.768385] INFO pktc_do_connect (pktc_post.h:19) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=49] PNIO sk_new: sk=0x7f3cd33ff358, fd=216 [2024-04-22 16:23:04.768381] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187313945245, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774184768260) [2024-04-22 16:23:04.768412] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774184568171, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:04.768480] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] write client fd succ, fd:216, gid:0x100000002, need_send_negotiation:1 [2024-04-22 16:23:04.768497] INFO eloop_regist (eloop.c:47) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO sock regist: 0x7f3cd33ff358 fd=216 [2024-04-22 16:23:04.768518] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO sock not ready: 0x7f3cd33ff358, fd=216 [2024-04-22 16:23:04.768526] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] sock regist: 0x7f3cfd8074d0 fd=216 [2024-04-22 16:23:04.768886] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] [ussl] client send negotiation message succ, fd:216, addr:172.23.88.129:52418, auth_method:NONE, gid:0x100000002 [2024-04-22 16:23:04.768971] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=43] [ussl] give back fd to origin epoll succ, client_fd:216, client_epfd:117, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:04.769032] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO sock connect OK: 0x7f3cd33ff358 fd:216:local:172.23.88.129:52418:remote:172.23.88.130:2882 [2024-04-22 16:23:04.776910] INFO [PALF] ack_config_log (palf_handle_impl.cpp:4098) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08DC33AC-0-0] [lt=112] ack_config_log success(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", msg_proposal_id=7, config_version={proposal_id:7, config_seq:8}) [2024-04-22 16:23:04.784713] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.785205] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.785800] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.786323] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184786248), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.786344] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=31000, remain_us=1414029, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.786969] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC90-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.786998] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC90-0-0] [lt=21] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.797112] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7E-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.797186] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7E-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:04.797226] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7E-0-0] [lt=36][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.800661] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=35][errcode=0] server is initiating(server_id=0, local_seq=14790, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.801826] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:2497, tid:10923}, {errcode:-5019, dropped:1362, tid:10899}]) [2024-04-22 16:23:04.802493] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=51][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774185798059, snapshot={val:1713774187411312889, v:0}) [2024-04-22 16:23:04.802570] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=73][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774185798059) [2024-04-22 16:23:04.802624] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=50][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[19, 20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774185798059, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:3, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:405, dyn_buf_header_offset:389, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10250}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:429, dyn_buf_header_offset:413, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10328}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10400}], op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f3cbde14310, valid:true, source:1, core:{version:{val:1713774187411312889, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184802366, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3cbde11fd0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3cbde11ff0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3cbde11f90, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3cbde121f0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.803118] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=490][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.803145] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=24][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25418, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3cd153d770, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240262, assoc_session_id:3221240262, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184802236, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184800013, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cbde09a28, valid:true, source:1, core:{version:{val:1713774187411312889, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184802366, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:3, 19, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3cbde11fd0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3cbde11ff0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3cbde11f90, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3cbde121f0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:0], access_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:405, dyn_buf_header_offset:389, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10250}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:429, dyn_buf_header_offset:413, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10328}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10400}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:0], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:405, dyn_buf_header_offset:389, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10250}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:429, dyn_buf_header_offset:413, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10328}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10400}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774185798059, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.803746] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=597][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.803781] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=31][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.803795] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=13][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.803807] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=10][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.803817] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=10][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.803828] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=10][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.803839] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=9][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=2) [2024-04-22 16:23:04.803863] WDIAG [SQL.EXE] inner_get_next_row (ob_direct_transmit_op.cpp:27) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=22][errcode=-4725] get next row from child failed(ret=-4725) [2024-04-22 16:23:04.803875] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=11][errcode=-4725] inner get next row failed(ret=-4725, type=14, op="PHY_DIRECT_TRANSMIT", op_id=1) [2024-04-22 16:23:04.803888] WDIAG [SQL.EXE] sync_send_result (ob_remote_executor_processor.cpp:283) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=11][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.803899] WDIAG [SQL.EXE] execute_remote_plan (ob_remote_executor_processor.cpp:469) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=9][errcode=-4725] sync send result failed(ret=-4725) [2024-04-22 16:23:04.803911] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=9][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.803924] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=13][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.803948] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=11] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd153d770, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240262, assoc_session_id:3221240262, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184802236, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184800013, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240262, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153d770}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.804049] WDIAG [SQL.EXE] process (ob_remote_executor_processor.cpp:990) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=89][errcode=-4725] execute remote plan failed(ret=-4725, local_tenant_schema_version=1, exec_ctx_.get_task_exec_ctx().get_query_tenant_begin_schema_version()=1) [2024-04-22 16:23:04.804240] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.807185] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.807259] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=62][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:45, local_retry_times:45, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:04.807310] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=30] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184761161, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184758954, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.807477] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=114][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.807513] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.807528] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:04.807577] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:04.807832] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=61][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774185641139, snapshot={val:1713774187411312890, v:0}) [2024-04-22 16:23:04.807883] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=49][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774185641139) [2024-04-22 16:23:04.807904] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=18][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[19, 20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774185641139, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:3, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f3cd47b6b90, valid:true, source:1, core:{version:{val:1713774187411312890, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184807785, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5415ab0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5415ad0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5415a70, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5415b20, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.808308] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=401][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.808334] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=24][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25419, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184806596, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184800013, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cff1ce058, valid:true, source:1, core:{version:{val:1713774187411312890, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184807785, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:3, 19, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5415ab0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5415ad0, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5415a70, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5415b20, count:4, allocator:0x7f3ce5406eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:0], access_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:0], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:549, dyn_buf_header_offset:533, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414148}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5414220}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:717, dyn_buf_header_offset:701, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce54142f8}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774185641139, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.808969] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=631][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.809002] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=28][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.809063] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=26][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.809096] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=30][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.809134] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=35][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.809161] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=25][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.809245] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=80][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=0) [2024-04-22 16:23:04.809281] WDIAG [SQL.EXE] get_next_row (ob_execute_result.cpp:147) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=32][errcode=-4725] get next row from operator failed(ret=-4725) [2024-04-22 16:23:04.809311] WDIAG [SQL] inner_get_next_row (ob_result_set.cpp:411) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=28][errcode=-4725] get next row from exec result failed(ret=-4725) [2024-04-22 16:23:04.809339] WDIAG [SERVER] open (ob_inner_sql_result.cpp:165) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=25][errcode=-4725] get_next_row failed(ret=-4725, has_tenant_resource()=true) [2024-04-22 16:23:04.809370] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=28][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:04.809404] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=46, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.809484] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=64] will sleep(sleep_us=46000, remain_us=931657, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.816866] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=11][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.816961] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=74] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.817024] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=57][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:04.817078] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=50][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:04.817132] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=51][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:04.817172] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=32][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:04.817204] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=29][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:04.817239] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=26][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:04.817271] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=29][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:04.817301] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=27][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:04.817330] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=27][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:04.817358] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=26][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:04.817388] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=27][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:04.817477] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=78][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:04.817511] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=30][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.817545] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=29][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.817575] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=27][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:04.817606] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=26][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.817640] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=30][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:04.817670] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=27][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.817713] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=38][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:04.817787] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=66][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.817818] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=28][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.817873] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=52][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:04.817929] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC6D-0-0] [lt=38][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:04.817981] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC6D-0-0] [lt=46][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:04.818035] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.818045] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=127] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:04.818114] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:04.818163] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:04.818217] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:04.818246] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.818268] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd1460050, start=1713774184815029, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.818368] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184814987), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.818698] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.819225] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.819683] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.819709] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=32000, remain_us=1380664, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.820333] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.820359] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.820370] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:04.820379] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=8][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:04.820395] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:04.820405] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=7][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:04.820417] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:04.820427] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:04.820452] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=24][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:04.820459] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=6][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:04.820465] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=5][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:04.820469] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=4][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:04.820474] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:04.820482] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:04.820488] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.820493] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.820498] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:04.820504] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.820509] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=5][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:04.820514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.820523] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=8][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:04.820533] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.820537] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.820542] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:04.820551] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC91-0-0] [lt=5][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:04.820557] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10479][EvtHisUpdTask][T0][YB42AC175881-000616AB09EFEC91-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:04.820568] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.820583] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:04.820596] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:04.820605] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:04.820611] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd45cc050, start=1713774184819699, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.820641] WDIAG [SHARE] process_task (ob_event_history_table_operator.cpp:324) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.820652] WDIAG [SHARE] process (ob_event_history_table_operator.cpp:170) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] process_task failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184819618), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), is_delete=false, ret=-5019, ret="OB_TABLE_NOT_EXIST", create_time_=1713774184819634) [2024-04-22 16:23:04.822966] INFO keepalive_check (time_wheel.c:76) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=39] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd33ff358,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:04.826277] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.826302] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=18] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-04-22 16:23:04.826313] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-04-22 16:23:04.826319] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-04-22 16:23:04.826326] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-04-22 16:23:04.826334] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:04.826338] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:04.826344] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-04-22 16:23:04.826348] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:04.826353] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:04.826359] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:04.826364] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:04.826368] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:04.826373] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:04.826379] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:04.826387] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:04.826391] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.826397] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.826401] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:04.826405] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-04-22 16:23:04.826410] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=5][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:04.826415] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.826426] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=9][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:04.826451] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.826457] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.826462] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:04.826471] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=5][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:04.826477] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCD-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-04-22 16:23:04.826483] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-04-22 16:23:04.826489] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:04.826494] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:04.826499] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f3ceeb84050, start=1713774184826201, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-04-22 16:23:04.826507] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:04.826512] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-04-22 16:23:04.826549] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:540) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-04-22 16:23:04.826604] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:666) [10796][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=55][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:04.827226] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774214727141, snapshot={val:1713774187411312891, v:0}) [2024-04-22 16:23:04.827247] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774214727141) [2024-04-22 16:23:04.827257] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774214727141, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:2, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], op_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], trans_desc:NULL, snapshot:{this:0x7f3cca606d70, valid:true, source:1, core:{version:{val:1713774187411312891, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184827204, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.827414] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=156][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.827423] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25420, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:0, need_switch_param:0, trans_desc:{this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184826420, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cd9453038, valid:true, source:1, core:{version:{val:1713774187411312891, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184827204, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:2, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], access_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:1, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774214727141, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.827648] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=223][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.827660] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.827665] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.827669] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.827673] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.827678] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.827683] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=0) [2024-04-22 16:23:04.827689] WDIAG [SQL.EXE] get_next_row (ob_execute_result.cpp:147) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] get next row from operator failed(ret=-4725) [2024-04-22 16:23:04.827694] WDIAG [SQL] inner_get_next_row (ob_result_set.cpp:411) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] get next row from exec result failed(ret=-4725) [2024-04-22 16:23:04.827698] WDIAG [SERVER] open (ob_inner_sql_result.cpp:165) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] get_next_row failed(ret=-4725, has_tenant_resource()=true) [2024-04-22 16:23:04.827704] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:04.827710] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.827722] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6] already timeout, do not need sleep(sleep_us=0, remain_us=29999419, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.827780] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.827804] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:04.827822] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184826420, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.827874] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=44][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.827881] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.827885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:04.827895] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:04.827962] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774214727141, snapshot={val:1713774187411312892, v:0}) [2024-04-22 16:23:04.827981] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774214727141) [2024-04-22 16:23:04.827991] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774214727141, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:2, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], op_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], trans_desc:NULL, snapshot:{this:0x7f3cca606d70, valid:true, source:1, core:{version:{val:1713774187411312892, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184827947, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.828156] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=164][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.828165] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25421, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184827536, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cd9453038, valid:true, source:1, core:{version:{val:1713774187411312892, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184827947, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:2, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], access_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:1, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774214727141, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.828356] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=190][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.828368] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.828372] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.828376] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.828381] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.828386] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.828391] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=0) [2024-04-22 16:23:04.828397] WDIAG [SQL.EXE] get_next_row (ob_execute_result.cpp:147) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] get next row from operator failed(ret=-4725) [2024-04-22 16:23:04.828401] WDIAG [SQL] inner_get_next_row (ob_result_set.cpp:411) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] get next row from exec result failed(ret=-4725) [2024-04-22 16:23:04.828406] WDIAG [SERVER] open (ob_inner_sql_result.cpp:165) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] get_next_row failed(ret=-4725, has_tenant_resource()=true) [2024-04-22 16:23:04.828412] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:04.828417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.828428] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6] will sleep(sleep_us=1000, remain_us=29998714, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.829539] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.829562] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:04.829578] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184827536, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.829635] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=51][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.829642] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.829647] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:04.829656] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:04.829715] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774214727141, snapshot={val:1713774187411312893, v:0}) [2024-04-22 16:23:04.829731] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774214727141) [2024-04-22 16:23:04.829740] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774214727141, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:2, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], op_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], trans_desc:NULL, snapshot:{this:0x7f3cca606d70, valid:true, source:1, core:{version:{val:1713774187411312893, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184829702, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.829910] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=169][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.829918] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25422, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184828634, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cd9453038, valid:true, source:1, core:{version:{val:1713774187411312893, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184829702, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:2, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], access_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:1, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774214727141, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.830108] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=188][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.830120] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.830125] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.830129] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.830133] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.830137] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.830140] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=0) [2024-04-22 16:23:04.830149] WDIAG [SQL.EXE] get_next_row (ob_execute_result.cpp:147) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] get next row from operator failed(ret=-4725) [2024-04-22 16:23:04.830154] WDIAG [SQL] inner_get_next_row (ob_result_set.cpp:411) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] get next row from exec result failed(ret=-4725) [2024-04-22 16:23:04.830166] WDIAG [SERVER] open (ob_inner_sql_result.cpp:165) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] get_next_row failed(ret=-4725, has_tenant_resource()=true) [2024-04-22 16:23:04.830171] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:04.830177] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.830188] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7] will sleep(sleep_us=2000, remain_us=29996953, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.832289] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.832313] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:04.832330] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184828634, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.832391] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.832397] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.832401] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:04.832410] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:04.832481] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774214727141, snapshot={val:1713774187411312894, v:0}) [2024-04-22 16:23:04.832499] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774214727141) [2024-04-22 16:23:04.832508] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774214727141, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:2, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], op_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], trans_desc:NULL, snapshot:{this:0x7f3cca606d70, valid:true, source:1, core:{version:{val:1713774187411312894, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184832468, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.832651] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=142][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.832657] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25423, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184831943, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cd9453038, valid:true, source:1, core:{version:{val:1713774187411312894, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184832468, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:2, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], access_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:1, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774214727141, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.832859] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=201][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.832871] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.832877] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.832883] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.832886] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.832891] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.832896] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=0) [2024-04-22 16:23:04.832903] WDIAG [SQL.EXE] get_next_row (ob_execute_result.cpp:147) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] get next row from operator failed(ret=-4725) [2024-04-22 16:23:04.832907] WDIAG [SQL] inner_get_next_row (ob_result_set.cpp:411) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3][errcode=-4725] get next row from exec result failed(ret=-4725) [2024-04-22 16:23:04.832911] WDIAG [SERVER] open (ob_inner_sql_result.cpp:165) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] get_next_row failed(ret=-4725, has_tenant_resource()=true) [2024-04-22 16:23:04.832916] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:04.832922] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=3, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.832933] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7] will sleep(sleep_us=3000, remain_us=29994208, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.836031] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.836055] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:3, local_retry_times:3, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:04.836072] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184831943, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.836118] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.836127] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.836133] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:04.836144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:04.836202] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774214727141, snapshot={val:1713774187411312896, v:0}) [2024-04-22 16:23:04.836218] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774214727141) [2024-04-22 16:23:04.836227] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774214727141, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:2, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], op_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], trans_desc:NULL, snapshot:{this:0x7f3cca606d70, valid:true, source:1, core:{version:{val:1713774187411312896, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184836189, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.836372] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=144][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.836381] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25425, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184835264, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cd9453038, valid:true, source:1, core:{version:{val:1713774187411312896, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184836189, snapshot_lsid:{id:1}, snapshot_ls_role:1, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:2, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3ce5445d90, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3ce5445db0, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3ce5445d50, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3ce5445e00, count:4, allocator:0x7f3ce5407eb0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], access_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:81, res_buf_off:565, dyn_buf_header_offset:549, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444158}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3ce5444308}], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:1, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3ce5444230}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774214727141, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.836586] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=204][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.836599] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.836605] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.836610] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.836615] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.836619] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.836624] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=0) [2024-04-22 16:23:04.836630] WDIAG [SQL.EXE] get_next_row (ob_execute_result.cpp:147) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] get next row from operator failed(ret=-4725) [2024-04-22 16:23:04.836641] WDIAG [SQL] inner_get_next_row (ob_result_set.cpp:411) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] get next row from exec result failed(ret=-4725) [2024-04-22 16:23:04.836647] WDIAG [SERVER] open (ob_inner_sql_result.cpp:165) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] get_next_row failed(ret=-4725, has_tenant_resource()=true) [2024-04-22 16:23:04.836653] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:04.836659] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=4, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.836670] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7] will sleep(sleep_us=4000, remain_us=29990471, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.840770] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.840796] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:4, local_retry_times:4, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:04.840814] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184835264, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.840870] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:04.840878] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.840961] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=5000, remain_us=29986181, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.841092] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFBF4-0-0] [lt=26][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774183740597}) [2024-04-22 16:23:04.844130] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=58][errcode=0] server is initiating(server_id=0, local_seq=14791, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.846069] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.846100] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184840004, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184818703, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.846207] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=6000, remain_us=29980935, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.846262] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=71][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774185798016, snapshot={val:1713774187411312899, v:0}) [2024-04-22 16:23:04.846300] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10952][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=2][errcode=0] server is initiating(server_id=0, local_seq=14792, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.846381] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=116][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774185798016) [2024-04-22 16:23:04.846407] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=24][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[19, 20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774185798016, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:3, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:405, dyn_buf_header_offset:389, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10250}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:429, dyn_buf_header_offset:413, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10328}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10400}], op_filters:NULL, trans_desc:NULL, snapshot:{this:0x7f3cbde14310, valid:true, source:1, core:{version:{val:1713774187411312899, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184846222, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3cbde11fd0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3cbde11ff0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3cbde11f90, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3cbde121f0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.846996] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=585][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.847017] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=20][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25428, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3cd153f690, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240263, assoc_session_id:3221240263, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184845587, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184845586, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cbde09a28, valid:true, source:1, core:{version:{val:1713774187411312899, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184846222, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:3, 19, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:3, 1, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:3, true, true, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3cbde11fd0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3cbde11ff0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3cbde11f90, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3cbde121f0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:0], access_exprs:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:405, dyn_buf_header_offset:389, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10250}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:429, dyn_buf_header_offset:413, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10328}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10400}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:0], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:3, {type:"T_REF_COLUMN", datum_meta:{type:"BIGINT", cs_type:"binary", scale:0, precision:20}, obj_meta:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:66, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:405, dyn_buf_header_offset:389, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10250}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:81, res_buf_off:429, dyn_buf_header_offset:413, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:97, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10328}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:162, res_buf_off:573, dyn_buf_header_offset:557, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10400}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774185798016, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.847797] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=775][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.847866] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=61][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.847917] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=47][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.847961] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=41][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.848008] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=44][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.848052] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=40][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.848111] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=54][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=2) [2024-04-22 16:23:04.848162] WDIAG [SQL.EXE] inner_get_next_row (ob_direct_transmit_op.cpp:27) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=47][errcode=-4725] get next row from child failed(ret=-4725) [2024-04-22 16:23:04.848207] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=40][errcode=-4725] inner get next row failed(ret=-4725, type=14, op="PHY_DIRECT_TRANSMIT", op_id=1) [2024-04-22 16:23:04.848255] WDIAG [SQL.EXE] sync_send_result (ob_remote_executor_processor.cpp:283) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=44][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.848300] WDIAG [SQL.EXE] execute_remote_plan (ob_remote_executor_processor.cpp:469) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=42][errcode=-4725] sync send result failed(ret=-4725) [2024-04-22 16:23:04.848241] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10952][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cf4641310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240264, assoc_session_id:3221240264, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184847843, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184847842, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc8afe98, id:3221240264, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cf4641310}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.848352] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=46][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.848407] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10952][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.848469] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=114][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.848519] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=29] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd153f690, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240263, assoc_session_id:3221240263, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184845587, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184845586, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240263, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153f690}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.848698] WDIAG [SQL.EXE] process (ob_remote_executor_processor.cpp:990) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=160][errcode=-4725] execute remote plan failed(ret=-4725, local_tenant_schema_version=1, exec_ctx_.get_task_exec_ctx().get_query_tenant_begin_schema_version()=1) [2024-04-22 16:23:04.848999] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=75] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.849882] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=65][errcode=0] server is initiating(server_id=0, local_seq=14793, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.851873] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=60][errcode=-4725] failed to check and get tablet(ret=-4725, ls_id={id:1}, tablet_id={id:1}, timeout=1713774214746077, snapshot={val:1713774187411312901, v:0}) [2024-04-22 16:23:04.851936] WDIAG [STORAGE] check_read_allowed_ (ob_access_service.cpp:561) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=61][errcode=-4725] failed to check replica allow to read(ret=-4725, tablet_id={id:1}, timeout=1713774214746077) [2024-04-22 16:23:04.852313] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.851961] WDIAG [SQL.DAS] open_op (ob_das_scan_op.cpp:329) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=23][errcode=-4725] fail to scan table(scan_param_={tablet_id:{id:1}, ls_id:{id:1}, column_ids:[20, 21], index_id:1, key_ranges:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], ss_key_ranges:[{"range":"table_id:1,group_idx:0,(MIN;MAX)"}], range_array_pos:[], timeout:1713774214746077, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}, sql_mode:4194304, reserved_cell_count:2, schema_version:0, query_begin_schema_version:1, limit_offset:{offset_:0, limit_:-1}, for_update:false, wait:-1, frozen_version:-1, is_get:false, output_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10450}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:421, dyn_buf_header_offset:405, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde102a0}], op_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3cbde10528}], trans_desc:NULL, snapshot:{this:0x7f3cbde144f0, valid:true, source:1, core:{version:{val:1713774187411312901, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184851832, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3cbde12190, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3cbde121b0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3cbde12150, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3cbde123b0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, sample_info:{method:0, percent:"1.000000000000000000e+02", seed:-1, table_id:0, scope:0, force_block:false}, need_scn:false, need_switch_param:false, fb_read_tx_uncommitted:false, external_file_format:}, external_file_location:""}, ret=-4725) [2024-04-22 16:23:04.852347] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184845587, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184843371, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.852410] WDIAG [SQL.DAS] start_das_task (ob_das_task.cpp:232) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=445][errcode=-4725] open das task op failed(ret=-4725) [2024-04-22 16:23:04.852493] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=7000, remain_us=29974648, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.852648] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.852428] WDIAG [SQL.DAS] execute_das_task (ob_data_access_service.cpp:98) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=17][errcode=-4725] start das task failed(ret=-4725, *task_wrapper.at(i)={parent:{tenant_id:1, task_id:25430, op_type:1, errcode:-4725, can_part_retry:0, task_started:1, in_part_retry:0, in_stmt_retry:1, need_switch_param:0, trans_desc:{this:0x7f3cd15415b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240265, assoc_session_id:3221240265, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184851258, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184849430, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, snapshot:{this:0x7f3cbde09a28, valid:true, source:1, core:{version:{val:1713774187411312901, v:0}, tx_id:{txid:0}, scn:0}, uncertain_bound:1713774184851832, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, tablet_id:{id:1}, ls_id:{id:1}, tablet_loc:{tablet_id:{id:1}, ls_id:{id:1}, server:"172.23.88.129:2882", in_retry:0, partition_id:1, first_level_part_id:18446744073709551615}, related_ctdefs:[cnt:0], related_rtdefs:[cnt:0], task_status:1, related_tablet_ids:[cnt:0], das_task_node:}, scan_ctdef:{ObDASBaseCtDef:{op_type:1}, ref_table_id:1, access_column_ids:[cnt:2, 20, 21], aggregate_column_ids:[cnt:0], group_by_column_ids:[cnt:0], schema_version:0, table_param:{table_id:1, output_projector:[cnt:2, 2, 3], aggregate_projector:[cnt:0], group_by_projector:[cnt:0], output_sel_mask:[cnt:2, false, true], pad_col_projector:[cnt:0], read_param_version:1, main_read_info:{schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, trans_col_index:-1, seq_read_column_count:3, max_col_index:7, is_oracle_mode:false, cols_index:{rowkey_mode:false, for_memtable:false, array:{ObIArray:[cnt:4, 0, 1, 2, 7], data:0x7f3cbde12190, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, memtable_cols_index:{rowkey_mode:false, for_memtable:true, array:{ObIArray:[cnt:4, 0, 1, 2, 5], data:0x7f3cbde121b0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}}, cols_desc:{ObIArray:[cnt:4, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0], data:0x7f3cbde12150, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, cg_idxs:{ObIArray:[cnt:0], data:null, count:0, allocator:null, capacity:0, init_cnt:0}, cols_extend:{ObIArray:[cnt:4, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}, {pack:0, min_max:0, sum:0}], data:0x7f3cbde123b0, count:4, allocator:0x7f3cd40562e0, capacity:4, init_cnt:4}, has_all_column_group:true, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, is_inited:true, is_oracle_mode:false}, cols_param:[{column_id:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0, accuracy:{length:-1, precision:20, scale:0}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:128, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:false, is_nullable_for_read:false, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}, {column_id:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0, accuracy:{length:65536, precision:-1, scale:-1}, orig_default_value:{"NULL":"NULL"}, cur_default_value:{"NULL":"NULL"}, is_nullable_for_write:true, is_nullable_for_read:true, is_gen_col:false, is_virtual_gen_col:false, is_gen_col_udf_expr:false, is_hidden:false, lob_chunk_size:262144}]}, use_lob_locator:false, rowid_version:0, rowid_projector:[cnt:0], enable_lob_locator_v2:true}, pd_expr_spec:{calc_exprs:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3cbde10528}], access_exprs:[cnt:2, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:2, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:1, frame_idx:1, datum_off:162, res_buf_off:709, dyn_buf_header_offset:693, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:178, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde10450}, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:421, dyn_buf_header_offset:405, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde102a0}], ext_file_column_exprs:[cnt:0], ext_column_convert_exprs:[cnt:0], max_batch_size:0, pushdown_filters:[cnt:1, {type:"T_OP_EQ", datum_meta:{type:"INT", cs_type:"binary", scale:0, precision:1}, obj_meta:{type:"INT", collation:"binary", coercibility:"NUMERIC"}, obj_datum_map:3, flag:202, eval_func:0x55963a74bf40, eval_batch_func:0x55963a511f40, inner_functions:null, inner_func_cnt:0, arg_cnt:2, parent_cnt:0, frame_idx:1, datum_off:243, res_buf_off:837, dyn_buf_header_offset:821, res_buf_len:8, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:259, eval_vector_func:0x5596438dbe40, local_session_var_id:-1, this:0x7f3cbde10528}], pd_storage_flag:{pd_flag:0}, trans_info_expr:NULL}, group_id_expr:NULL, result_output:[cnt:1, {type:"T_REF_COLUMN", datum_meta:{type:"VARCHAR", cs_type:"utf8mb4_general_ci", scale:-1, precision:-1}, obj_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"IMPLICIT"}, obj_datum_map:1, flag:130, eval_func:null, eval_batch_func:null, inner_functions:null, inner_func_cnt:0, arg_cnt:0, parent_cnt:0, frame_idx:1, datum_off:0, res_buf_off:421, dyn_buf_header_offset:405, res_buf_len:128, eval_flags_off:0, pvt_skip_off:0, expr_ctx_id:4294967295, extra:0, batch_idx_mask:0, vector_header_off:16, eval_vector_func:null, local_session_var_id:-1, this:0x7f3cbde102a0}], is_get:false, is_external_table:false, external_files:[cnt:0], external_file_format_str:{str:""}, external_file_location:{str:""}, trans_info_expr:NULL, ir_scan_type:0}, scan_rtdef:{ObDASBaseRtDef:{op_type:1}, tenant_schema_version:1, limit_param:{offset_:0, limit_:-1}, need_scn:false, force_refresh_lc:false, frozen_version:-1, fb_snapshot:{val:18446744073709551615, v:3}, fb_read_tx_uncommitted:false, timeout_ts:1713774214746077, tx_lock_timeout:-1, sql_mode:4194304, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}, scan_range:[{"range":"table_id:1,group_idx:0,({"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MIN,MIN;{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},MAX,MAX)"}], result:{}, scan_flag:{scan_order:1, daily_merge:0, rmmb_optimize:0, whole_macro_scan:0, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:0, is_need_feedback:1, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, enable_rich_format:0, is_for_foreign_key_check:0, is_mds_query:0, is_select_follower:0, enable_lob_prefetch:0, reserved:0}}) [2024-04-22 16:23:04.853051] WDIAG [SQL.DAS] execute_all_task (ob_das_ref.cpp:268) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=619][errcode=-4725] failed to execute aggregated das task(ret=-4725, ret="OB_TABLET_NOT_EXIST", aggregated_task={server:"172.23.88.129:2882", high_priority_tasks_.get_size():0, tasks_.get_size():0, failed_tasks_.get_size():1, success_tasks_.get_size():0}, async=true) [2024-04-22 16:23:04.853091] WDIAG [SQL.ENG] do_table_scan (ob_table_scan_op.cpp:2234) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=34][errcode=-4725] execute all das scan task failed(ret=-4725) [2024-04-22 16:23:04.853108] WDIAG [SQL.ENG] do_init_before_get_row (ob_table_scan_op.cpp:1478) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=16][errcode=-4725] fail to do table scan(ret=-4725) [2024-04-22 16:23:04.853126] WDIAG [SQL.ENG] inner_get_next_row_for_tsc (ob_table_scan_op.cpp:2007) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=17][errcode=-4725] failed to init before get row(ret=-4725) [2024-04-22 16:23:04.853137] WDIAG [SQL.ENG] inner_get_next_row_implement (ob_table_scan_op.cpp:1994) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=11][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.853147] WDIAG [SQL.ENG] inner_get_next_row (ob_table_scan_op.cpp:2981) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=10][errcode=-4725] get next row failed(ret=-4725) [2024-04-22 16:23:04.853158] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=9][errcode=-4725] inner get next row failed(ret=-4725, type=3, op="PHY_TABLE_SCAN", op_id=2) [2024-04-22 16:23:04.853174] WDIAG [SQL.EXE] inner_get_next_row (ob_direct_transmit_op.cpp:27) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=15][errcode=-4725] get next row from child failed(ret=-4725) [2024-04-22 16:23:04.853185] WDIAG [SQL.ENG] get_next_row (ob_operator.cpp:1162) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=11][errcode=-4725] inner get next row failed(ret=-4725, type=14, op="PHY_DIRECT_TRANSMIT", op_id=1) [2024-04-22 16:23:04.853198] WDIAG [SQL.EXE] sync_send_result (ob_remote_executor_processor.cpp:283) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=12][errcode=-4725] failed to get next row(ret=-4725) [2024-04-22 16:23:04.853212] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.853225] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=12][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.853246] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=10] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15415b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240265, assoc_session_id:3221240265, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184851258, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184849430, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240265, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15415b0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.853422] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.853617] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.853972] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.854746] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.854816] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=33000, remain_us=1345557, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.855788] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=73] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.855886] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10888][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=29][errcode=0] server is initiating(server_id=0, local_seq=14794, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.855885] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184806596, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184800013, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.856278] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=47000, remain_us=884863, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.856944] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=8][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.857024] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=61] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.857056] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:04.857094] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=35][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:04.857113] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:04.857131] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:04.857144] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:04.857161] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:04.857173] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:04.857185] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=10][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:04.857196] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=10][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:04.857207] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:04.857218] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:04.857239] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:04.857252] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.857276] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=21][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.857289] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:04.857302] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.857316] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:04.857328] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.857362] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=28][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:04.857398] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=28][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.857416] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.857431] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:04.857566] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC92-0-0] [lt=69][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:04.857623] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10479][EvtHisUpdTask][T0][YB42AC175881-000616AB09EFEC92-0-0] [lt=49][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:04.857660] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.857716] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:04.857776] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:04.857795] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:04.857810] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cc12b0050, start=1713774184854804, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.857896] WDIAG [SHARE] process_task (ob_event_history_table_operator.cpp:324) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-5019] execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.857929] WDIAG [SHARE] process (ob_event_history_table_operator.cpp:170) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-5019] process_task failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184854657), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), is_delete=false, ret=-5019, ret="OB_TABLE_NOT_EXIST", create_time_=1713774184854698) [2024-04-22 16:23:04.858002] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10888][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=31][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.858117] WDIAG push_back (ob_fixed_array.h:71) [10888][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.858147] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10888][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=28][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.858177] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10888][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=17] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f3070, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240266, assoc_session_id:3221240266, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184857851, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184855674, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240266, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3ccb8f3070}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.858536] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10888][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.859600] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.859634] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184852368, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184851257, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.859760] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=8000, remain_us=29967382, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.861815] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=40][errcode=0] server is initiating(server_id=0, local_seq=14795, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.862536] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.862560] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=23][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.862581] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=15] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15434d0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240267, assoc_session_id:3221240267, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184862172, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184861093, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240267, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15434d0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.862707] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.867026] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=13][errcode=0] server is initiating(server_id=0, local_seq=14796, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.867856] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.867885] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184858930, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184855674, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.867932] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.867946] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=13][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.868001] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=29959140, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.867960] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=8] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15453f0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240268, assoc_session_id:3221240268, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184867576, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184867706, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240268, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15453f0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.868133] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.869110] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:04.873328] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=33][errcode=0] server is initiating(server_id=0, local_seq=14797, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.874361] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.874402] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=39][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.874430] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=18] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1547310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240269, assoc_session_id:3221240269, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184873204, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184872057, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240269, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1547310}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.874682] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.877105] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.877143] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184867576, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184867706, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.877262] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=10000, remain_us=29949879, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.880801] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10955][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=3][errcode=0] server is initiating(server_id=0, local_seq=14798, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.882912] WDIAG push_back (ob_fixed_array.h:71) [10955][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.882977] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10955][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=61][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.883011] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10955][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=21] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cc567f850, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240270, assoc_session_id:3221240270, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184882149, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184879916, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc8b8ce8, id:3221240270, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cc567f850}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.883457] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10955][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.886189] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886256] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=64] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886279] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886299] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886313] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886325] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886337] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886348] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886360] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886372] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886384] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886403] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886415] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886427] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886471] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.886483] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:04.887389] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.887449] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184876542, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184872057, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.887582] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] get number(ret=0, service_type_=0, range=1, base_id=1713774184887580923, start_id=1713774187411312913, end_id=1713774187411312914) [2024-04-22 16:23:04.887613] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774184887580923, last_request_ts=1713774184786162399, time_delta=101418524, last_gts=1713774187400013204, gts=1713774187411312914, gts_delta=11299709, compensation_value=10141852) [2024-04-22 16:23:04.887667] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=11000, remain_us=29939475, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.889280] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.890066] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=35][errcode=0] server is initiating(server_id=0, local_seq=14799, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.890096] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.890462] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10953][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=0] server is initiating(server_id=0, local_seq=14800, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.890733] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.891161] WDIAG push_back (ob_fixed_array.h:71) [10953][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.891182] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10953][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=20][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.891195] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10953][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=8] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2332890, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240272, assoc_session_id:3221240272, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184891092, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184889981, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc8661f8, id:3221240272, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd2332890}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.891326] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10953][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.891404] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.891494] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1308879, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.891978] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.892025] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=45][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.892071] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=29] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1549230, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240271, assoc_session_id:3221240271, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184891092, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184891091, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240271, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1549230}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.892397] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.893567] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=25][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.893678] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=87] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.893712] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=31][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:04.893732] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:04.893839] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=104][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:04.893880] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=30][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:04.893903] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=21][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:04.893930] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=16][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:04.893957] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=23][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:04.893992] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=32][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:04.894006] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=13][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:04.894017] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:04.894030] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:04.894070] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=29][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:04.894083] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.894098] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.894110] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:04.894123] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.894138] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:04.894151] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.894177] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:04.894203] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.894221] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.894250] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=27][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:04.894281] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC93-0-0] [lt=16][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:04.894340] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10479][EvtHisUpdTask][T0][YB42AC175881-000616AB09EFEC93-0-0] [lt=51][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:04.894369] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.894432] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:04.894557] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=119][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:04.894594] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:04.894628] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd148e050, start=1713774184891481, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.894697] WDIAG [SHARE] process_task (ob_event_history_table_operator.cpp:324) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.894730] WDIAG [SHARE] process (ob_event_history_table_operator.cpp:170) [10479][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-5019] process_task failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184891317), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), is_delete=false, ret=-5019, ret="OB_TABLE_NOT_EXIST", create_time_=1713774184891369) [2024-04-22 16:23:04.898778] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.898818] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184886606, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184879916, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.898976] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=12000, remain_us=29928166, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.899498] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=86][errcode=0] server is initiating(server_id=0, local_seq=14801, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.901153] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.901203] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=48][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.901242] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154b150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240273, assoc_session_id:3221240273, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184901019, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184898813, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240273, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154b150}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.901557] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.903502] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=42] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.903595] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184855675, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184855674, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.903903] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=48000, remain_us=837237, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.910424] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=49][errcode=0] server is initiating(server_id=0, local_seq=14802, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.911105] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.911146] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184898814, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184891091, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.911267] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=29915874, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.911336] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=68][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.911381] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.911394] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=12][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.911411] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=10] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154d070, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240274, assoc_session_id:3221240274, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184911033, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184909911, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240274, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154d070}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.911611] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.921230] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=86] PNIO [ratelimit] time: 1713774184921228, bytes: 65862938, bw: 0.257285 MB/s, add_ts: 1006467, add_bytes: 271528 [2024-04-22 16:23:04.921729] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=30][errcode=0] server is initiating(server_id=0, local_seq=14803, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.922575] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.922598] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=22][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.922619] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=15] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154ef90, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240275, assoc_session_id:3221240275, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184922161, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184921033, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240275, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154ef90}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.922749] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.924377] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.924411] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184911033, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184909911, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.924599] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=14000, remain_us=29902543, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.924780] INFO keepalive_check (time_wheel.c:76) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd33fe048,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:04.926691] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.927310] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.927984] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.928689] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184928615), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.928768] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1271606, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.930834] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=25][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.930890] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=37] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.930940] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=46][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:04.930960] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:04.930979] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:04.930997] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:04.931009] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:04.931026] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:04.931038] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:04.931050] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=11][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:04.931062] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=11][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:04.931072] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:04.931085] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:04.931106] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:04.931128] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=19][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.931143] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:04.931155] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:04.931168] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184928615), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:04.931183] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:04.931194] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184928615), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:04.931220] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:04.931249] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.931267] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:04.931281] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=13][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:04.931404] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC94-0-0] [lt=43][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184928615), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:04.931497] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10479][EvtHisUpdTask][T0][YB42AC175881-000616AB09EFEC94-0-0] [lt=88][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184928615), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:04.933929] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=40][errcode=0] server is initiating(server_id=0, local_seq=14804, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.934933] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.934971] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=36][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.934999] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=18] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1550eb0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240276, assoc_session_id:3221240276, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184934521, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184933394, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240276, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1550eb0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.935236] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.935316] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=36][errcode=0] server is initiating(server_id=0, local_seq=14805, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.936608] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.936646] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=36][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.936672] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=18] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1552dd0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240277, assoc_session_id:3221240277, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184935629, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184933394, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240277, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1552dd0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.936888] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.938740] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.938812] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184924406, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184921033, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.939013] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=29888129, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.939034] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] sock regist: 0x7f3cfd8726b0 fd=195 [2024-04-22 16:23:04.939098] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=49] [ussl] accept new connection, fd:195, src_addr:172.23.88.130:55312 [2024-04-22 16:23:04.939199] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:195, src_addr:172.23.88.130:55312 [2024-04-22 16:23:04.939243] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] PNIO dispatch fd to certain group, fd:195, gid:0x100000000 [2024-04-22 16:23:04.939385] INFO pkts_sk_init (pkts_sk_factory.h:23) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO set pkts_sk_t sock_id s=0x7f3cd1aac9b8, s->id=524285 [2024-04-22 16:23:04.939432] INFO pkts_sk_new (pkts_sk_factory.h:51) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO sk_new: s=0x7f3cd1aac9b8 [2024-04-22 16:23:04.939564] INFO eloop_regist (eloop.c:47) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=56] PNIO sock regist: 0x7f3cd1aac9b8 fd=195 [2024-04-22 16:23:04.939617] INFO on_accept (listenfd.c:39) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO accept new connection, ns=0x7f3cd1aac9b8, fd=fd:195:local:172.23.88.129:2882:remote:172.23.88.130:55312 [2024-04-22 16:23:04.939736] WDIAG listenfd_handle_event (listenfd.c:71) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:04.939773] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1482) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F58-0-0] [lt=27] receive register_req from acceptor, ignore(self="172.23.88.129:2882", palf_id=1, child={server:"172.23.88.130:2882", region:"default_region", register_time_us:1713774184935158, keepalive_ts:-1}) [2024-04-22 16:23:04.939818] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1501) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F58-0-0] [lt=43] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", child={server:"172.23.88.130:2882", region:"default_region", register_time_us:1713774184935158, keepalive_ts:-1}, is_to_leader=true) [2024-04-22 16:23:04.942418] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=36][errcode=0] server is initiating(server_id=0, local_seq=14806, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.944371] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.944422] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=48][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.944494] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=58] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1558b30, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240278, assoc_session_id:3221240278, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184943415, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184941195, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240278, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1558b30}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.944693] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=68] PNIO [ratelimit] time: 1713774184944690, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007577, add_bytes: 0 [2024-04-22 16:23:04.944796] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.948054] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10896][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=118][errcode=0] server is initiating(server_id=0, local_seq=14807, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.950248] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10896][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=62][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.950341] WDIAG push_back (ob_fixed_array.h:71) [10896][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.950363] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10896][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=21][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.950388] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10896][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=14] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cf4645150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240279, assoc_session_id:3221240279, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184950105, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184946762, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc86f048, id:3221240279, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cf4645150}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.950762] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10896][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.952113] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=47] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.952209] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184903227, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184898813, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.952636] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=49000, remain_us=788504, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:04.954177] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.954245] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184937822, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184933394, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.954540] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=16000, remain_us=29872602, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.964921] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=60][errcode=0] server is initiating(server_id=0, local_seq=14808, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.964997] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.965706] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.966369] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:04.966741] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.966791] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=48][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.966831] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=24] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1556c10, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240280, assoc_session_id:3221240280, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184965951, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184965952, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240280, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1556c10}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.967074] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774184966956), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:04.967136] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=36000, remain_us=1233238, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:04.967275] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.967843] INFO [LIB] stat (utility.h:1165) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=97, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:04.967925] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="LEADER", total_group_log_cnt=5, avg_log_batch_cnt=1, total_group_log_size=601, avg_group_log_size=120) [2024-04-22 16:23:04.968023] INFO [PALF] submit_log (palf_handle_impl.cpp:453) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, append size=601) [2024-04-22 16:23:04.968069] INFO [LIB] stat (utility.h:1165) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT APPEND COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=174, this=0x7f3cd2d6d1d0, extra_info="id:1") [2024-04-22 16:23:04.968485] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:04.968549] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187313945245, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774184968473) [2024-04-22 16:23:04.968599] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774184768476, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:04.969304] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC95-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:04.969376] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC95-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:04.970767] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.970887] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184953464, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184946762, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.971413] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=17000, remain_us=29855729, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:04.972750] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT WRITE LOG COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=3292, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:04.972794] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=601) [2024-04-22 16:23:04.972836] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=60, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:04.972903] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4084) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="172.23.88.129:2882", server="172.23.88.129:2882", update_func={old_end_lsn:{lsn:358454}, new_end_lsn:{lsn:358576}, old_advance_time_us:1713774183870220, new_ack_time_us:1713774184972896, advance delay(us):1102676}) [2024-04-22 16:23:04.973010] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=96] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=845) [2024-04-22 16:23:04.973045] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=40, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:04.975497] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4084) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DA2-0-0] [lt=28][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self="172.23.88.129:2882", server="172.23.88.131:2882", update_func={old_end_lsn:{lsn:358454}, new_end_lsn:{lsn:358576}, old_advance_time_us:1713774183872578, new_ack_time_us:1713774184975493, advance delay(us):1102915}) [2024-04-22 16:23:04.975744] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DA2-0-0] [lt=79] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=14, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:04.975805] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DA2-0-0] [lt=59] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=5625, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:04.975830] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DA2-0-0] [lt=22] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=4, avg_log_gen_to_submit_time=115, avg_log_submit_to_flush_time=3365, avg_log_submit_to_slide_time=5509) [2024-04-22 16:23:04.980971] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=68][errcode=0] server is initiating(server_id=0, local_seq=14809, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.982903] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.982905] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10893][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=78][errcode=0] server is initiating(server_id=0, local_seq=14810, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:04.982957] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=51][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.982996] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd155aa50, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240281, assoc_session_id:3221240281, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184982633, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184981507, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240281, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155aa50}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.983294] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.984982] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10893][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=74][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:04.985083] WDIAG push_back (ob_fixed_array.h:71) [10893][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:04.985104] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10893][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=20][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:04.985130] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10893][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=14] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb92f230, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240282, assoc_session_id:3221240282, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184984881, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184981507, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc880ce8, id:3221240282, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3ccb92f230}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:04.985493] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10893][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:04.988756] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=144] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:04.988855] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184970455, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184968436, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:04.989161] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] get number(ret=0, service_type_=0, range=1, base_id=1713774184989158268, start_id=1713774187421454788, end_id=1713774187421454789) [2024-04-22 16:23:04.989224] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=58][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774184989158268, last_request_ts=1713774184887580923, time_delta=101577345, last_gts=1713774187411312914, gts=1713774187421454789, gts_delta=10141874, compensation_value=10157734) [2024-04-22 16:23:04.989433] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36] will sleep(sleep_us=18000, remain_us=29837709, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.001267] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=61][errcode=0] server is initiating(server_id=0, local_seq=14811, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.001964] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=43] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.002084] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184952365, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184946762, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.002286] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-8001, dropped:695, tid:10436}]) [2024-04-22 16:23:05.002534] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=50000, remain_us=738607, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.003212] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.003267] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=52][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.003331] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=50][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:36, local_retry_times:36, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.003304] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=24] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd155c970, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240283, assoc_session_id:3221240283, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185002729, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185000477, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240283, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155c970}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.003419] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=78][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.003536] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=113][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.003560] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.003610] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=20][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:05.003645] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.004758] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=48][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1196200) [2024-04-22 16:23:05.004819] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=55][errcode=-8001] [2024-04-22 16:23:05.004900] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.005167] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [10532][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=47] runTimer to refresh locality_info(ret=0) [2024-04-22 16:23:05.005575] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1196200) [2024-04-22 16:23:05.005616] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=37][errcode=-8001] [2024-04-22 16:23:05.005651] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=14] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.006296] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1196200) [2024-04-22 16:23:05.006332] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=32][errcode=-8001] [2024-04-22 16:23:05.006362] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.006509] INFO [STORAGE] runTimerTask (ob_ddl_heart_beat_task.cpp:49) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12] send to rs all task status succeed [2024-04-22 16:23:05.006600] INFO [STORAGE.BLKMGR] mark_and_sweep (ob_block_manager.cpp:1008) [10529][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=71] no block alloc/free, no need to mark blocks(ret=0, mark_info.count()=0) [2024-04-22 16:23:05.006670] INFO [STORAGE.BLKMGR] mark_and_sweep (ob_block_manager.cpp:1037) [10529][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=54] finish once mark and sweep(ret=0, alloc_num=4, mark_info.count()=0, marker_status={total_block_count:0, reserved_block_count:0, linked_block_count:0, tmp_file_count:0, data_block_count:0, shared_data_block_count:0, index_block_count:0, ids_block_count:0, disk_block_count:0, bloomfiter_count:0, hold_count:4, pending_free_count:0, free_count:76794, shared_meta_block_count:0, mark_cost_time:111, sweep_cost_time:0, start_time:"2024-04-22 16:23:05.6545", last_end_time:"2024-04-22 16:23:05.6666", hold_info:nothing}, map_cnt=4) [2024-04-22 16:23:05.007089] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1196200) [2024-04-22 16:23:05.007142] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=49][errcode=-8001] [2024-04-22 16:23:05.007166] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:05.007180] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=4] TIMER THREAD STAT: (thread_id=10430, task_cnt=269, avg_time=1966) [2024-04-22 16:23:05.007184] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDD-0-0] [lt=25][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.007214] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=33] TIMER THREAD STAT: (thread_id=10431, task_cnt=271, avg_time=799) [2024-04-22 16:23:05.007228] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=13] TIMER THREAD STAT: (thread_id=10404, task_cnt=158, avg_time=601) [2024-04-22 16:23:05.007239] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10541, task_cnt=56, avg_time=3) [2024-04-22 16:23:05.007230] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDD-0-0] [lt=30] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:05.007250] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=11] TIMER THREAD STAT: (thread_id=10529, task_cnt=55, avg_time=3551) [2024-04-22 16:23:05.007260] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10406, task_cnt=18, avg_time=42) [2024-04-22 16:23:05.007259] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDD-0-0] [lt=25][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.007271] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10608, task_cnt=18, avg_time=48) [2024-04-22 16:23:05.007267] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=29] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185007198), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.007281] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10717, task_cnt=1799, avg_time=188) [2024-04-22 16:23:05.007290] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:05.007292] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10808, task_cnt=2711, avg_time=2) [2024-04-22 16:23:05.007303] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10807, task_cnt=67, avg_time=10) [2024-04-22 16:23:05.007304] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:05.007313] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=10720, task_cnt=103, avg_time=28093) [2024-04-22 16:23:05.007315] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=9][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:05.007327] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:05.007324] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10407, task_cnt=11, avg_time=104) [2024-04-22 16:23:05.007342] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=37, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:05.007360] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=36] TIMER THREAD STAT: (thread_id=10739, task_cnt=55, avg_time=4) [2024-04-22 16:23:05.007399] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=25] will sleep(sleep_us=37000, remain_us=1192992, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.007404] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=42] TIMER THREAD STAT: (thread_id=10829, task_cnt=52, avg_time=49755) [2024-04-22 16:23:05.007421] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=16] TIMER THREAD STAT: (thread_id=10862, task_cnt=228, avg_time=9) [2024-04-22 16:23:05.007416] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=1] load region(ret=-5019, locality_info={version:0, local_region:"", local_zone:"", local_idc:"", local_zone_type:3, local_zone_status:3, locality_region_array:[], locality_zone_array:[]}) [2024-04-22 16:23:05.007437] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=15] TIMER THREAD STAT: (thread_id=10845, task_cnt=27, avg_time=187) [2024-04-22 16:23:05.007525] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=85] TIMER THREAD STAT: (thread_id=10796, task_cnt=4, avg_time=12961478) [2024-04-22 16:23:05.007538] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=12] TIMER THREAD STAT: (thread_id=10723, task_cnt=11, avg_time=32) [2024-04-22 16:23:05.007548] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=10738, task_cnt=11, avg_time=8609) [2024-04-22 16:23:05.007558] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10801, task_cnt=11, avg_time=2497) [2024-04-22 16:23:05.007568] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10843, task_cnt=11, avg_time=3865) [2024-04-22 16:23:05.007578] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10844, task_cnt=11, avg_time=416) [2024-04-22 16:23:05.007409] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1060) [10817][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=26] replay status is not follower(min_unreplayed_lsn={lsn:356581}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774182065832, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:356581}, next_to_submit_scn_:{val:1713774177880657608, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:356459}, end_lsn:{lsn:356581}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774177880657607, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774177880657607, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:1991013568, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774177880657607, v:0}}}}}) [2024-04-22 16:23:05.007588] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10846, task_cnt=11, avg_time=188) [2024-04-22 16:23:05.007599] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10847, task_cnt=11, avg_time=137) [2024-04-22 16:23:05.007597] INFO [CLOG] operator() (ob_log_replay_service.cpp:1492) [10817][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=186] get_replay_process success(id={id:1}, replayed_log_size=356581, unreplayed_log_size=0) [2024-04-22 16:23:05.007609] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10409, task_cnt=6, avg_time=5264) [2024-04-22 16:23:05.007620] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10817, task_cnt=5, avg_time=244) [2024-04-22 16:23:05.007638] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=17] TIMER THREAD STAT: (thread_id=10492, task_cnt=30, avg_time=25612) [2024-04-22 16:23:05.007649] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10475, task_cnt=22, avg_time=3) [2024-04-22 16:23:05.007659] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=10410, task_cnt=56, avg_time=2600) [2024-04-22 16:23:05.007669] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=11011, task_cnt=55, avg_time=38) [2024-04-22 16:23:05.007679] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=11016, task_cnt=55, avg_time=8533) [2024-04-22 16:23:05.007673] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:156) [10817][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=21] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-04-22 16:23:05.007689] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=10520, task_cnt=52, avg_time=175469) [2024-04-22 16:23:05.007699] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10901, task_cnt=26, avg_time=78412) [2024-04-22 16:23:05.007709] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10519, task_cnt=11, avg_time=182501) [2024-04-22 16:23:05.007674] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=74] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.007719] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10532, task_cnt=6, avg_time=106) [2024-04-22 16:23:05.007730] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10542, task_cnt=6, avg_time=53) [2024-04-22 16:23:05.007740] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=10521, task_cnt=6, avg_time=224) [2024-04-22 16:23:05.007750] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=10856, task_cnt=2, avg_time=484) [2024-04-22 16:23:05.007760] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10857, task_cnt=2, avg_time=155) [2024-04-22 16:23:05.007809] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=47] TIMER THREAD STAT: (thread_id=10536, task_cnt=1, avg_time=31) [2024-04-22 16:23:05.007819] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=10] TIMER THREAD STAT: (thread_id=10751, task_cnt=1, avg_time=190) [2024-04-22 16:23:05.007837] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=17] TIMER THREAD STAT: (thread_id=10854, task_cnt=1, avg_time=1072) [2024-04-22 16:23:05.007750] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774184988215, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774184987115, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.007847] INFO dump (ob_timer_monitor.cpp:200) [10542][ObTimer][T0][Y0-0000000000000000-0-0] [lt=9] TIMER THREAD STAT: (thread_id=10530, task_cnt=0, avg_time=0) [2024-04-22 16:23:05.008061] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=29819081, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.008910] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [10521][DumpLSLoc][T0][YB42AC175881-000616AB0AFFEBDD-0-0] [lt=85] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774185007664, replica_locations:[{server:"172.23.88.129:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}, {server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2024-04-22 16:23:05.009120] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC96-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.009193] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC96-0-0] [lt=55] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.018886] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:05.020262] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=42][errcode=0] server is initiating(server_id=0, local_seq=14812, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.022159] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.022212] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=50][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.022256] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=28] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd155e890, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240284, assoc_session_id:3221240284, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185020983, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185019871, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240284, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155e890}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.022644] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.027290] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=54] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.027390] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185007433, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185006206, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.027774] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=20000, remain_us=29799368, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.028701] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] server is initiating(server_id=0, local_seq=14813, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.030707] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.030742] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=33][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.030779] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=23] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15607b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240285, assoc_session_id:3221240285, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185029940, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185027728, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240285, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15607b0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.031087] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.034379] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=15420, clean_start_pos=3774870, clean_num=125829) [2024-04-22 16:23:05.040495] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=48][errcode=0] server is initiating(server_id=0, local_seq=14814, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.042939] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.043006] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=63][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.043047] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=27] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15242d0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240286, assoc_session_id:3221240286, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185042136, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185039920, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240286, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15242d0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.043493] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.044540] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:37, local_retry_times:37, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.044612] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=62][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.044641] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.044654] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.044736] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=63][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:05.045927] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=98][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1155159) [2024-04-22 16:23:05.045991] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=58][errcode=-8001] [2024-04-22 16:23:05.046045] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.046675] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=30][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1155159) [2024-04-22 16:23:05.046724] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=45][errcode=-8001] [2024-04-22 16:23:05.046758] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.047522] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=34][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1155159) [2024-04-22 16:23:05.047567] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=41][errcode=-8001] [2024-04-22 16:23:05.047604] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.047657] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10962][T1_L0_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=69][errcode=0] server is initiating(server_id=0, local_seq=14815, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.048068] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.048195] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185026633, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185019871, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.048340] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=43][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1155159) [2024-04-22 16:23:05.048360] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18][errcode=-8001] [2024-04-22 16:23:05.048373] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=6][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:05.048449] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185048390), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.048468] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:05.048477] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:05.048488] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:05.048494] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=5][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:05.048502] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=7][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=38, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:05.048517] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8] will sleep(sleep_us=38000, remain_us=1151856, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.048624] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=21000, remain_us=29778518, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.048675] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10962][T1_L0_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=16][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.048726] WDIAG push_back (ob_fixed_array.h:71) [10962][T1_L0_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.048746] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10962][T1_L0_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=18][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.048762] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10962][T1_L0_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=9] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cf85c67b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240287, assoc_session_id:3221240287, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185047623, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185047623, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc877e98, id:3221240287, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cf85c67b0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.048949] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10962][T1_L0_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.050305] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC97-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.050389] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC97-0-0] [lt=62] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.052798] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=76] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.052893] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185001608, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185000477, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.053252] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=51000, remain_us=687888, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.061711] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC41-0-0] [lt=50][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.061806] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC41-0-0] [lt=78] table not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, table_name.ptr()="data_size:25, data:5F5F616C6C5F6469736B5F696F5F63616C6962726174696F6E", ret=-5019) [2024-04-22 16:23:05.061837] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC41-0-0] [lt=28][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.062290] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=66][errcode=0] server is initiating(server_id=0, local_seq=14816, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.064111] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.064174] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=61][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.064206] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=20] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15261f0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240288, assoc_session_id:3221240288, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185063818, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185061666, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240288, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15261f0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.064572] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.069324] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:05.069860] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.069958] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185047623, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185047623, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.070323] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=22000, remain_us=29756819, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.077309] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10948][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=2][errcode=0] server is initiating(server_id=0, local_seq=14817, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.079332] WDIAG push_back (ob_fixed_array.h:71) [10948][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.079391] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10948][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=56][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.079482] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10948][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=76] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15a8890, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240289, assoc_session_id:3221240289, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185078222, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185077111, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240289, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15a8890}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.079839] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10948][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.084341] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10959][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=68][errcode=0] server is initiating(server_id=0, local_seq=14818, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.085403] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10959][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=39][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.085471] WDIAG push_back (ob_fixed_array.h:71) [10959][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.085482] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10959][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=11][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.085496] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10959][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=7] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cc5ee4890, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240290, assoc_session_id:3221240290, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185085015, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185083897, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc8661f8, id:3221240290, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cc5ee4890}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.085658] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10959][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.086650] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:38, local_retry_times:38, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.086762] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=103][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.086784] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.086797] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.086822] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:05.088031] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=33][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1113245) [2024-04-22 16:23:05.088088] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=51][errcode=-8001] [2024-04-22 16:23:05.088139] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=20] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.088726] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1113245) [2024-04-22 16:23:05.088772] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=41][errcode=-8001] [2024-04-22 16:23:05.088819] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.089622] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1113245) [2024-04-22 16:23:05.089677] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=49][errcode=-8001] [2024-04-22 16:23:05.089715] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=14] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.090382] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=99][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1113245) [2024-04-22 16:23:05.090428] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=42][errcode=-8001] [2024-04-22 16:23:05.090491] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=50][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:05.090620] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=48] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185090543), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.090665] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=40][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:05.090685] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:05.090732] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=44][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:05.090762] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=27][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:05.090782] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=39, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:05.090816] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=20] will sleep(sleep_us=39000, remain_us=1109558, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.092518] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.092529] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC98-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.092585] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC98-0-0] [lt=38] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.092589] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185069357, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185061666, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.092869] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] get number(ret=0, service_type_=0, range=1, base_id=1713774185092866045, start_id=1713774187431612539, end_id=1713774187431612540) [2024-04-22 16:23:05.092919] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774185092866045, last_request_ts=1713774184989158268, time_delta=103707777, last_gts=1713774187421454789, gts=1713774187431612540, gts_delta=10157750, compensation_value=10370777) [2024-04-22 16:23:05.093020] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=23000, remain_us=29734122, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.104526] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=56] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.104626] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185052984, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185047623, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.105014] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=52000, remain_us=636127, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.106393] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE0-0-0] [lt=31][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.106501] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE0-0-0] [lt=92] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-04-22 16:23:05.106544] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE0-0-0] [lt=38][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.106737] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:108) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE0-0-0] [lt=0] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:05.106942] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=65][errcode=0] server is initiating(server_id=0, local_seq=14819, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.108382] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE0-0-0] [lt=40][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.108484] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE0-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-04-22 16:23:05.108762] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:301) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE0-0-0] [lt=0] cal_all_part_disk_default_percentage succ(data_dir="/home/creality/oceanbase/store/sstable", clog_dir="/home/creality/oceanbase/store/clog", shared_mode=false, data_disk_total_size=205110317056, data_disk_default_percentage=90, clog_disk_total_size=85856358400, clog_disk_default_percentage=90) [2024-04-22 16:23:05.108838] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:337) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE0-0-0] [lt=73] decide disk size finished(suggested_disk_size=81604378624, suggested_disk_percentage=0, default_disk_percentage=90, total_space=85856358400, disk_size=81604378624) [2024-04-22 16:23:05.108859] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE0-0-0] [lt=20] get_log_disk_info_in_config(suggested_data_disk_size=161061273600, suggested_clog_disk_size=81604378624, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=81604378624, log_disk_percentage=0, total_log_disk_size=85856358400) [2024-04-22 16:23:05.108879] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE0-0-0] [lt=16] try_resize success(ret=0, log_disk_size=81604378624, total_log_disk_size=85856358400, this={dir::"/home/creality/oceanbase/store/clog/log_pool", dir_fd:14, meta_fd:15, log_pool_meta:{curr_total_size:81604378624, next_total_size:81604378624, status:0}, min_block_id:2, max_block_id:1216, min_log_disk_size_for_all_tenants_:4294967296, is_inited:true}) [2024-04-22 16:23:05.108884] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.108926] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:132) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE0-0-0] [lt=45] refresh tenant config(tenants=[], ret=-5019) [2024-04-22 16:23:05.108927] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=40][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.108978] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=33] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1528110, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240291, assoc_session_id:3221240291, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185108191, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185107099, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240291, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1528110}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.109344] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.116255] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=42] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.116379] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185091656, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185089464, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.116794] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=24000, remain_us=29710349, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.121224] INFO keepalive_check (time_wheel.c:76) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3d01ba4048,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:05.124333] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=33] === Run print tenant memory usage task === [2024-04-22 16:23:05.124485] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=49] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 579,035,136 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 192,438,272 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 115,343,360 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 12,582,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 33,783,808 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 201,011,200 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:05.125262] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=52] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 192,438,272 used= 181,620,464 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,477,760 used= 31,458,304 count= 1 avg_used= 31,458,304 block_cnt= 1 chunk_cnt= 1 mod=ash_list [MEMORY] hold= 22,761,472 used= 22,699,200 count= 9 avg_used= 2,522,133 block_cnt= 9 chunk_cnt= 8 mod=IoControl [MEMORY] hold= 16,883,712 used= 15,771,688 count= 229 avg_used= 68,872 block_cnt= 229 chunk_cnt= 31 mod=[T]ObSessionDIB [MEMORY] hold= 12,873,728 used= 12,856,352 count= 1 avg_used= 12,856,352 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 9,457,664 used= 9,437,784 count= 1 avg_used= 9,437,784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckInteChe [MEMORY] hold= 5,513,216 used= 5,502,336 count= 21 avg_used= 262,016 block_cnt= 21 chunk_cnt= 9 mod=ObTxDesc [MEMORY] hold= 5,331,904 used= 5,240,872 count= 49 avg_used= 106,956 block_cnt= 25 chunk_cnt= 11 mod=PartitTableTask [MEMORY] hold= 5,267,456 used= 5,190,496 count= 8 avg_used= 648,812 block_cnt= 8 chunk_cnt= 3 mod=KvstCachWashStr [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashPxBlooFilKe [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=PxP2PDhMgrKey [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBucTenComMo [MEMORY] hold= 4,307,328 used= 4,278,711 count= 3 avg_used= 1,426,237 block_cnt= 3 chunk_cnt= 3 mod=SqlDtlMgr [MEMORY] hold= 4,231,936 used= 4,203,008 count= 6 avg_used= 700,501 block_cnt= 5 chunk_cnt= 3 mod=BaseLogWriter [MEMORY] hold= 4,214,880 used= 4,194,328 count= 2 avg_used= 2,097,164 block_cnt= 2 chunk_cnt= 2 mod=SerFuncRegHT [MEMORY] hold= 3,276,800 used= 3,214,848 count= 11 avg_used= 292,258 block_cnt= 11 chunk_cnt= 7 mod=DedupQueue [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdUnitMa [MEMORY] hold= 2,167,040 used= 2,129,912 count= 7 avg_used= 304,273 block_cnt= 7 chunk_cnt= 5 mod=FixedQueue [MEMORY] hold= 2,114,080 used= 2,031,616 count= 1,026 avg_used= 1,980 block_cnt= 209 chunk_cnt= 3 mod=TabletLSMap [MEMORY] hold= 2,037,344 used= 1,915,008 count= 468 avg_used= 4,091 block_cnt= 243 chunk_cnt= 3 mod=CharsetInit [MEMORY] hold= 1,843,200 used= 1,638,400 count= 25 avg_used= 65,536 block_cnt= 25 chunk_cnt= 14 mod=[T]char [MEMORY] hold= 1,591,232 used= 1,560,960 count= 21 avg_used= 74,331 block_cnt= 20 chunk_cnt= 9 mod=TenantCtxAlloca [MEMORY] hold= 1,581,056 used= 1,572,904 count= 1 avg_used= 1,572,904 block_cnt= 1 chunk_cnt= 1 mod=IdConnMap [MEMORY] hold= 1,523,712 used= 1,507,118 count= 5 avg_used= 301,423 block_cnt= 5 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 1,452,608 used= 1,434,880 count= 44 avg_used= 32,610 block_cnt= 44 chunk_cnt= 20 mod=CommonArray [MEMORY] hold= 1,408,800 used= 1,388,000 count= 4 avg_used= 347,000 block_cnt= 4 chunk_cnt= 3 mod=Omt [MEMORY] hold= 1,286,144 used= 1,270,296 count= 2 avg_used= 635,148 block_cnt= 2 chunk_cnt= 2 mod=PxResMgr [MEMORY] hold= 1,140,864 used= 1,105,088 count= 557 avg_used= 1,984 block_cnt= 145 chunk_cnt= 24 mod=CreateContext [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=ConcurHashMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=TsSourceInfoMap [MEMORY] hold= 933,888 used= 897,024 count= 5 avg_used= 179,404 block_cnt= 5 chunk_cnt= 5 mod=LightyQueue [MEMORY] hold= 933,888 used= 917,576 count= 2 avg_used= 458,788 block_cnt= 2 chunk_cnt= 1 mod=CACHE_INST_MAP [MEMORY] hold= 827,472 used= 773,160 count= 13 avg_used= 59,473 block_cnt= 12 chunk_cnt= 8 mod=HashBucket [MEMORY] hold= 704,512 used= 691,200 count= 2 avg_used= 345,600 block_cnt= 2 chunk_cnt= 2 mod=LDBlockBitMap [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=EventTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=GEleTimer [MEMORY] hold= 622,592 used= 619,008 count= 1 avg_used= 619,008 block_cnt= 1 chunk_cnt= 1 mod=SysTaskStatus [MEMORY] hold= 614,400 used= 556,560 count= 15 avg_used= 37,104 block_cnt= 15 chunk_cnt= 8 mod=InneSqlConnPool [MEMORY] hold= 573,440 used= 304,920 count= 35 avg_used= 8,712 block_cnt= 35 chunk_cnt= 13 mod=[T]ObDSActionAr [MEMORY] hold= 512,000 used= 507,904 count= 64 avg_used= 7,936 block_cnt= 64 chunk_cnt= 2 mod=ModulePageAlloc [MEMORY] hold= 491,264 used= 441,808 count= 271 avg_used= 1,630 block_cnt= 116 chunk_cnt= 13 mod=tg [MEMORY] hold= 458,752 used= 457,856 count= 7 avg_used= 65,408 block_cnt= 7 chunk_cnt= 6 mod=ObTsResponseTas [MEMORY] hold= 448,000 used= 444,416 count= 56 avg_used= 7,936 block_cnt= 56 chunk_cnt= 13 mod=SessionInfoHash [MEMORY] hold= 425,984 used= 420,032 count= 1 avg_used= 420,032 block_cnt= 1 chunk_cnt= 1 mod=CompEventMgr [MEMORY] hold= 403,648 used= 378,880 count= 258 avg_used= 1,468 block_cnt= 54 chunk_cnt= 2 mod=LSLocationMap [MEMORY] hold= 401,408 used= 393,256 count= 1 avg_used= 393,256 block_cnt= 1 chunk_cnt= 1 mod=TablStorStatMgr [MEMORY] hold= 368,896 used= 330,944 count= 7 avg_used= 47,277 block_cnt= 7 chunk_cnt= 3 mod=Rpc [MEMORY] hold= 327,680 used= 325,960 count= 5 avg_used= 65,192 block_cnt= 5 chunk_cnt= 5 mod=LatchStat [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=register_tasks [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=register_task [MEMORY] hold= 262,144 used= 261,632 count= 4 avg_used= 65,408 block_cnt= 4 chunk_cnt= 4 mod=AsyncTaskQueue [MEMORY] hold= 229,376 used= 224,000 count= 1 avg_used= 224,000 block_cnt= 1 chunk_cnt= 1 mod=BGTMonitor [MEMORY] hold= 221,184 used= 212,992 count= 1 avg_used= 212,992 block_cnt= 1 chunk_cnt= 1 mod=TSWorker [MEMORY] hold= 221,184 used= 214,432 count= 1 avg_used= 214,432 block_cnt= 1 chunk_cnt= 1 mod=CompSuggestMgr [MEMORY] hold= 212,992 used= 196,624 count= 2 avg_used= 98,312 block_cnt= 2 chunk_cnt= 1 mod=DRTaskMap [MEMORY] hold= 212,992 used= 204,928 count= 1 avg_used= 204,928 block_cnt= 1 chunk_cnt= 1 mod=TenantMutilAllo [MEMORY] hold= 212,992 used= 196,624 count= 2 avg_used= 98,312 block_cnt= 2 chunk_cnt= 2 mod=DdlQue [MEMORY] hold= 203,872 used= 202,200 count= 26 avg_used= 7,776 block_cnt= 26 chunk_cnt= 6 mod=HashNode [MEMORY] hold= 180,224 used= 172,064 count= 1 avg_used= 172,064 block_cnt= 1 chunk_cnt= 1 mod=TenantMBList [MEMORY] hold= 163,840 used= 147,792 count= 2 avg_used= 73,896 block_cnt= 2 chunk_cnt= 1 mod=HashNodNexWaiMa [MEMORY] hold= 163,504 used= 155,416 count= 2 avg_used= 77,708 block_cnt= 2 chunk_cnt= 2 mod=OB_DISK_REP [MEMORY] hold= 160,624 used= 159,272 count= 21 avg_used= 7,584 block_cnt= 21 chunk_cnt= 2 mod=TenaSpaTabIdSet [MEMORY] hold= 155,648 used= 153,152 count= 3 avg_used= 51,050 block_cnt= 3 chunk_cnt= 2 mod=IO_MGR [MEMORY] hold= 155,648 used= 147,624 count= 1 avg_used= 147,624 block_cnt= 1 chunk_cnt= 1 mod=UsrRuleMap [MEMORY] hold= 144,880 used= 143,656 count= 19 avg_used= 7,560 block_cnt= 19 chunk_cnt= 2 mod=SysTableNameMap [MEMORY] hold= 139,264 used= 131,776 count= 1 avg_used= 131,776 block_cnt= 1 chunk_cnt= 1 mod=GtsTaskQueue [MEMORY] hold= 137,920 used= 134,912 count= 11 avg_used= 12,264 block_cnt= 11 chunk_cnt= 3 mod=SeArray [MEMORY] hold= 124,416 used= 122,688 count= 27 avg_used= 4,544 block_cnt= 27 chunk_cnt= 11 mod=[T]ObTraceEvent [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=TenaSchForCacMa [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=MemMgrForLiboMa [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=MemMgrMap [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=RefrFullScheMap [MEMORY] hold= 115,616 used= 114,656 count= 15 avg_used= 7,643 block_cnt= 15 chunk_cnt= 2 mod=HashNodeConfCon [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=NonPartTenMap [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=IndNameMap [MEMORY] hold= 114,336 used= 106,088 count= 2 avg_used= 53,044 block_cnt= 2 chunk_cnt= 2 mod=RetryCtrl [MEMORY] hold= 106,496 used= 98,312 count= 1 avg_used= 98,312 block_cnt= 1 chunk_cnt= 1 mod=TmpFileManager [MEMORY] hold= 106,496 used= 86,408 count= 5 avg_used= 17,281 block_cnt= 5 chunk_cnt= 2 mod=HashBuckConfCon [MEMORY] hold= 88,000 used= 56,704 count= 10 avg_used= 5,670 block_cnt= 9 chunk_cnt= 5 mod=BucketLock [MEMORY] hold= 81,920 used= 74,064 count= 2 avg_used= 37,032 block_cnt= 2 chunk_cnt= 2 mod=io_trace_map [MEMORY] hold= 79,328 used= 70,816 count= 8 avg_used= 8,852 block_cnt= 8 chunk_cnt= 5 mod=MallocInfoMap [MEMORY] hold= 73,728 used= 65,600 count= 1 avg_used= 65,600 block_cnt= 1 chunk_cnt= 1 mod=TCREF [MEMORY] hold= 73,728 used= 69,664 count= 1 avg_used= 69,664 block_cnt= 1 chunk_cnt= 1 mod=SuperBlockBuffe [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=ObLSLocation [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=KVCACHE_HAZARD [MEMORY] hold= 65,136 used= 63,096 count= 2 avg_used= 31,548 block_cnt= 2 chunk_cnt= 2 mod=ScheCacSysCacMa [MEMORY] hold= 64,000 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=RsEventQueue [MEMORY] hold= 57,344 used= 55,368 count= 1 avg_used= 55,368 block_cnt= 1 chunk_cnt= 1 mod=DepInfoTaskQ [MEMORY] hold= 56,128 used= 27,136 count= 377 avg_used= 71 block_cnt= 222 chunk_cnt= 22 mod=Coro [MEMORY] hold= 49,152 used= 32,768 count= 2 avg_used= 16,384 block_cnt= 2 chunk_cnt= 1 mod=CACHE_TNT_LST [MEMORY] hold= 49,152 used= 37,032 count= 3 avg_used= 12,344 block_cnt= 3 chunk_cnt= 2 mod=ReferedMap [MEMORY] hold= 49,152 used= 36,912 count= 2 avg_used= 18,456 block_cnt= 2 chunk_cnt= 2 mod=HashBuckSysConf [MEMORY] hold= 47,744 used= 38,400 count= 19 avg_used= 2,021 block_cnt= 15 chunk_cnt= 6 mod=TGTimer [MEMORY] hold= 45,760 used= 45,056 count= 11 avg_used= 4,096 block_cnt= 11 chunk_cnt= 6 mod=LinearHashMapCn [MEMORY] hold= 45,760 used= 45,056 count= 11 avg_used= 4,096 block_cnt= 11 chunk_cnt= 6 mod=LinearHashMapDi [MEMORY] hold= 41,264 used= 37,264 count= 2 avg_used= 18,632 block_cnt= 2 chunk_cnt= 1 mod=TaskRunnerSer [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdPoolMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SqlLoadData [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=KillSessMapBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=ClientSessBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=ObLongopsMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdConfMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucNamConMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HasBucConRefCoM [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucConPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucNamPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucSerUniMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucTenPooMa [MEMORY] hold= 40,960 used= 40,136 count= 1 avg_used= 40,136 block_cnt= 1 chunk_cnt= 1 mod=CommSysVarFac [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HasBucSerMigUnM [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucPooUniMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=T3MTBMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DDLSpeedCtrl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SessHoldMapBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TmpFileStoreMap [MEMORY] hold= 29,024 used= 26,512 count= 35 avg_used= 757 block_cnt= 25 chunk_cnt= 13 mod=ObGuard [MEMORY] hold= 24,944 used= 20,784 count= 2 avg_used= 10,392 block_cnt= 2 chunk_cnt= 2 mod=SqlNio [MEMORY] hold= 24,672 used= 24,264 count= 6 avg_used= 4,044 block_cnt= 6 chunk_cnt= 2 mod=ScheObSchemAren [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=FuncRuleMap [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=ServerCkptSlogH [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=GrpIdNameMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=leakMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=GrpNameIdMap [MEMORY] hold= 24,224 used= 21,672 count= 2 avg_used= 10,836 block_cnt= 2 chunk_cnt= 1 mod=SchemaStatuMap [MEMORY] hold= 23,904 used= 5,472 count= 228 avg_used= 24 block_cnt= 142 chunk_cnt= 24 mod=[T]MemoryContex [MEMORY] hold= 21,504 used= 20,352 count= 16 avg_used= 1,272 block_cnt= 16 chunk_cnt= 2 mod=IO_GROUP_MAP [MEMORY] hold= 18,416 used= 18,144 count= 4 avg_used= 4,536 block_cnt= 4 chunk_cnt= 3 mod=DeviceMng [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=HasBucTimZonInM [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MemDumpMap [MEMORY] hold= 16,384 used= 10,592 count= 1 avg_used= 10,592 block_cnt= 1 chunk_cnt= 1 mod=TenCompProgMgr [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ServerLogPool [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TbltRefreshMap [MEMORY] hold= 16,384 used= 8,992 count= 1 avg_used= 8,992 block_cnt= 1 chunk_cnt= 1 mod=IO_HEALTH [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=GenSchemVersMap [MEMORY] hold= 16,384 used= 12,296 count= 1 avg_used= 12,296 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 16,384 used= 9,336 count= 1 avg_used= 9,336 block_cnt= 1 chunk_cnt= 1 mod=InnerLobHash [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=LinkArray [MEMORY] hold= 16,032 used= 15,904 count= 2 avg_used= 7,952 block_cnt= 2 chunk_cnt= 1 mod=UpgProcSet [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=CommSysVarDefVa [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=SqlSession [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=MacroIdList [MEMORY] hold= 11,712 used= 8,944 count= 43 avg_used= 208 block_cnt= 26 chunk_cnt= 4 mod=Scheduler [MEMORY] hold= 10,816 used= 9,984 count= 13 avg_used= 768 block_cnt= 13 chunk_cnt= 4 mod=timer [MEMORY] hold= 9,472 used= 8,384 count= 16 avg_used= 524 block_cnt= 4 chunk_cnt= 2 mod=RpcBuffer [MEMORY] hold= 8,384 used= 8,104 count= 4 avg_used= 2,026 block_cnt= 3 chunk_cnt= 2 mod=TenantInfo [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PrepareOk [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ObTsTenantInfoN [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RetryTask [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=AddrList [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=Node [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SchemaRowKey [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PlanVaIdx [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=followerRenew [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=acceptOK [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerBlacklist [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerRegioMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerIdcMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerCidMap [MEMORY] hold= 7,968 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=LuaHandler [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HasNodTzInfM [MEMORY] hold= 7,776 used= 5,832 count= 27 avg_used= 216 block_cnt= 12 chunk_cnt= 8 mod=ObFuture [MEMORY] hold= 6,704 used= 3,840 count= 40 avg_used= 96 block_cnt= 30 chunk_cnt= 9 mod=PThread [MEMORY] hold= 5,424 used= 5,288 count= 2 avg_used= 2,644 block_cnt= 2 chunk_cnt= 2 mod=DeadLock [MEMORY] hold= 5,376 used= 5,248 count= 2 avg_used= 2,624 block_cnt= 1 chunk_cnt= 1 mod=RootContext [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=RebuildCtx [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=SqlPx [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmReq [MEMORY] hold= 4,592 used= 3,426 count= 16 avg_used= 214 block_cnt= 13 chunk_cnt= 9 mod=EventReHolder [MEMORY] hold= 3,888 used= 3,816 count= 1 avg_used= 3,816 block_cnt= 1 chunk_cnt= 1 mod=RecScheHisMap [MEMORY] hold= 3,888 used= 3,816 count= 1 avg_used= 3,816 block_cnt= 1 chunk_cnt= 1 mod=RemMasterMap [MEMORY] hold= 3,680 used= 3,360 count= 5 avg_used= 672 block_cnt= 5 chunk_cnt= 4 mod=[T]ObWarningBuf [MEMORY] hold= 3,264 used= 3,200 count= 1 avg_used= 3,200 block_cnt= 1 chunk_cnt= 1 mod=TenantTZ [MEMORY] hold= 2,768 used= 2,704 count= 1 avg_used= 2,704 block_cnt= 1 chunk_cnt= 1 mod=LoggerAlloc [MEMORY] hold= 2,544 used= 1,392 count= 16 avg_used= 87 block_cnt= 15 chunk_cnt= 8 mod=ObFunction [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=StorageS3 [MEMORY] hold= 1,728 used= 1,536 count= 3 avg_used= 512 block_cnt= 3 chunk_cnt= 2 mod=TntResourceMgr [MEMORY] hold= 1,600 used= 1,536 count= 1 avg_used= 1,536 block_cnt= 1 chunk_cnt= 1 mod=LobManager [MEMORY] hold= 1,504 used= 1,432 count= 1 avg_used= 1,432 block_cnt= 1 chunk_cnt= 1 mod=GtsRequestRpc [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=T3MQSyncLock [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=TsMgr [MEMORY] hold= 1,472 used= 1,400 count= 1 avg_used= 1,400 block_cnt= 1 chunk_cnt= 1 mod=GtiRequestRpc [MEMORY] hold= 1,408 used= 1,344 count= 1 avg_used= 1,344 block_cnt= 1 chunk_cnt= 1 mod=SchemaService [MEMORY] hold= 1,392 used= 1,328 count= 1 avg_used= 1,328 block_cnt= 1 chunk_cnt= 1 mod=GtsRpcProxy [MEMORY] hold= 1,392 used= 1,328 count= 1 avg_used= 1,328 block_cnt= 1 chunk_cnt= 1 mod=GtiRpcProxy [MEMORY] hold= 1,360 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=TENANT_PLAN_MAP [MEMORY] hold= 1,360 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IO_CHANNEL_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=INGRESS_MAP [MEMORY] hold= 1,184 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=memdumpqueue [MEMORY] hold= 1,168 used= 1,092 count= 1 avg_used= 1,092 block_cnt= 1 chunk_cnt= 1 mod=LogRegionMap [MEMORY] hold= 768 used= 704 count= 1 avg_used= 704 block_cnt= 1 chunk_cnt= 1 mod=ScheMgrCacheMap [MEMORY] hold= 640 used= 280 count= 5 avg_used= 56 block_cnt= 5 chunk_cnt= 1 mod=Log [MEMORY] hold= 528 used= 264 count= 3 avg_used= 88 block_cnt= 3 chunk_cnt= 2 mod=DestKAState [MEMORY] hold= 432 used= 272 count= 2 avg_used= 136 block_cnt= 2 chunk_cnt= 2 mod=unknown [MEMORY] hold= 288 used= 84 count= 3 avg_used= 28 block_cnt= 2 chunk_cnt= 2 mod=KeepAliveServer [MEMORY] hold= 240 used= 48 count= 3 avg_used= 16 block_cnt= 3 chunk_cnt= 3 mod=CreateEntity [MEMORY] hold= 224 used= 160 count= 1 avg_used= 160 block_cnt= 1 chunk_cnt= 1 mod=OccamTimeGuard [MEMORY] hold= 128 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=PxTargetMgr [MEMORY] hold= 80 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=Compressor [MEMORY] hold= 80 used= 7 count= 1 avg_used= 7 block_cnt= 1 chunk_cnt= 1 mod=SqlExpr [MEMORY] hold= 181,620,464 used= 178,203,526 count= 4,460 avg_used= 39,955 mod=SUMMARY [2024-04-22 16:23:05.125638] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=278] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 6,291,456 used= 2,658,560 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,600,016 used= 675,052 count= 12,703 avg_used= 53 block_cnt= 209 chunk_cnt= 2 mod=Buffer [MEMORY] hold= 1,005,024 used= 704,055 count= 3,307 avg_used= 212 block_cnt= 200 chunk_cnt= 3 mod=glibc_malloc [MEMORY] hold= 53,520 used= 37,729 count= 229 avg_used= 164 block_cnt= 29 chunk_cnt= 1 mod=S3SDK [MEMORY] hold= 2,658,560 used= 1,416,836 count= 16,239 avg_used= 87 mod=SUMMARY [2024-04-22 16:23:05.125791] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=92] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 115,343,360 used= 109,412,352 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 109,412,352 used= 109,208,832 count= 212 avg_used= 515,136 block_cnt= 212 chunk_cnt= 55 mod=CoStack [MEMORY] hold= 109,412,352 used= 109,208,832 count= 212 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:05.125884] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=64] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 6,291,456 used= 3,882,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,882,960 used= 3,662,624 count= 265 avg_used= 13,821 block_cnt= 47 chunk_cnt= 3 mod=OB_TEST2_PCODE [MEMORY] hold= 3,882,960 used= 3,662,624 count= 265 avg_used= 13,821 mod=SUMMARY [2024-04-22 16:23:05.125946] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=32] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 12,582,912 used= 12,484,608 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,323,072 used= 8,318,976 count= 4 avg_used= 2,079,744 block_cnt= 4 chunk_cnt= 4 mod=Logger [MEMORY] hold= 4,161,536 used= 4,159,488 count= 2 avg_used= 2,079,744 block_cnt= 2 chunk_cnt= 2 mod=ErrorLogger [MEMORY] hold= 12,484,608 used= 12,478,464 count= 6 avg_used= 2,079,744 mod=SUMMARY [2024-04-22 16:23:05.126162] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=34] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 33,783,808 used= 30,848,016 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 25,482,256 used= 25,332,400 count= 45 avg_used= 562,942 block_cnt= 23 chunk_cnt= 12 mod=DEFAULT [MEMORY] hold= 2,760,704 used= 2,750,496 count= 21 avg_used= 130,976 block_cnt= 21 chunk_cnt= 4 mod=PKTS_INBUF [MEMORY] hold= 1,835,008 used= 1,833,664 count= 14 avg_used= 130,976 block_cnt= 14 chunk_cnt= 4 mod=PKTC_INBUF [MEMORY] hold= 245,760 used= 244,320 count= 15 avg_used= 16,288 block_cnt= 15 chunk_cnt= 4 mod=SERVER_CTX_CHUN [MEMORY] hold= 196,608 used= 195,456 count= 12 avg_used= 16,288 block_cnt= 12 chunk_cnt= 3 mod=SERVER_RESP_CHU [MEMORY] hold= 196,608 used= 195,456 count= 12 avg_used= 16,288 block_cnt= 12 chunk_cnt= 3 mod=CLIENT_CB_CHUNK [MEMORY] hold= 131,072 used= 130,304 count= 8 avg_used= 16,288 block_cnt= 8 chunk_cnt= 2 mod=CLIENT_REQ_CHUN [MEMORY] hold= 30,848,016 used= 30,682,096 count= 127 avg_used= 241,591 mod=SUMMARY [2024-04-22 16:23:05.126569] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=40] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 11,292,672 used= 9,760,320 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 7,098,368 used= 7,078,824 count= 1 avg_used= 7,078,824 block_cnt= 1 chunk_cnt= 1 mod=SchemaIdVersion [MEMORY] hold= 2,088,768 used= 2,087,680 count= 2 avg_used= 1,043,840 block_cnt= 2 chunk_cnt= 2 mod=TenantSchemMgr [MEMORY] hold= 360,448 used= 327,680 count= 4 avg_used= 81,920 block_cnt= 4 chunk_cnt= 1 mod=SchemaMgrCache [MEMORY] hold= 110,208 used= 102,397 count= 5 avg_used= 20,479 block_cnt= 3 chunk_cnt= 1 mod=SchemaSysCache [MEMORY] hold= 31,872 used= 31,616 count= 4 avg_used= 7,904 block_cnt= 4 chunk_cnt= 1 mod=ScheTenaInfoVec [MEMORY] hold= 16,320 used= 16,064 count= 4 avg_used= 4,016 block_cnt= 4 chunk_cnt= 1 mod=SchemaSysVariab [MEMORY] hold= 15,936 used= 15,808 count= 2 avg_used= 7,904 block_cnt= 2 chunk_cnt= 1 mod=ScheTablInfoVec [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheIndeNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheTablIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=HiddenTblNames [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheRoutIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheRoutNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheTablNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=SchePackIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=SchePackNameMap [MEMORY] hold= 1,536 used= 1,024 count= 8 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSeCompo [MEMORY] hold= 1,152 used= 768 count= 6 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSePolic [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=ScheUdtNameMap [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=ScheUdtIdMap [MEMORY] hold= 1,152 used= 768 count= 6 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSeLabel [MEMORY] hold= 800 used= 512 count= 4 avg_used= 128 block_cnt= 3 chunk_cnt= 1 mod=SchemaSynonym [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=DBLINK_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabSeUserLe [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 4 chunk_cnt= 1 mod=DIRECTORY_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_POLICY_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlSqlMap [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_GROUP_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaProfile [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_CONTEXT_MGR [MEMORY] hold= 656 used= 584 count= 1 avg_used= 584 block_cnt= 1 chunk_cnt= 1 mod=TenSchMemMgrFoL [MEMORY] hold= 656 used= 584 count= 1 avg_used= 584 block_cnt= 1 chunk_cnt= 1 mod=TenaScheMemMgr [MEMORY] hold= 416 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheDataNameMap [MEMORY] hold= 416 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=MockFkParentTab [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaSecurAudi [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheAuxVpNameVe [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheForKeyNamMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaContext [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheConsNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlIdMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchePriTabPriMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchePriObjPriMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheTrigIdMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaTablespac [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheTrigNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaUdf [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaSequence [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaKeystore [MEMORY] hold= 9,760,320 used= 9,691,189 count= 134 avg_used= 72,322 mod=SUMMARY [2024-04-22 16:23:05.126734] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=94] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 201,011,200 used= 199,004,208 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 134,238,288 used= 134,217,736 count= 2 avg_used= 67,108,868 block_cnt= 2 chunk_cnt= 2 mod=CACHE_MAP_BKT [MEMORY] hold= 18,051,072 used= 18,033,840 count= 1 avg_used= 18,033,840 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MB_HANDLE [MEMORY] hold= 17,329,216 used= 8,968,960 count= 1,093 avg_used= 8,205 block_cnt= 1,093 chunk_cnt= 9 mod=StorageLoggerM [MEMORY] hold= 16,806,784 used= 16,786,208 count= 3 avg_used= 5,595,402 block_cnt= 3 chunk_cnt= 2 mod=FixeSizeBlocAll [MEMORY] hold= 6,311,936 used= 6,291,472 count= 1 avg_used= 6,291,472 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MAP_LOCK [MEMORY] hold= 3,709,312 used= 3,295,616 count= 114 avg_used= 28,908 block_cnt= 55 chunk_cnt= 4 mod=OccamThreadPool [MEMORY] hold= 2,543,616 used= 2,523,256 count= 1 avg_used= 2,523,256 block_cnt= 1 chunk_cnt= 1 mod=TenantConfig [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=BlockMap [MEMORY] hold= 5,984 used= 1,088 count= 68 avg_used= 16 block_cnt= 2 chunk_cnt= 2 mod=ConfigChecker [MEMORY] hold= 199,004,208 used= 190,126,112 count= 1,284 avg_used= 148,073 mod=SUMMARY [2024-04-22 16:23:05.126802] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=63][errcode=0] server is initiating(server_id=0, local_seq=14820, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.126863] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=63] [MEMORY] tenant: 508, limit: 1,073,741,824 hold: 40,427,520 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 8,970,240 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 31,457,280 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:05.126984] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=45] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 8,970,240 used= 7,859,136 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4,423,680 used= 4,132,320 count= 60 avg_used= 68,872 block_cnt= 60 chunk_cnt= 3 mod=[T]ObSessionDIB [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 28,800 used= 24,960 count= 60 avg_used= 416 block_cnt= 7 chunk_cnt= 2 mod=OMT_Worker [MEMORY] hold= 16,320 used= 7,680 count= 120 avg_used= 64 block_cnt= 7 chunk_cnt= 2 mod=Coro [MEMORY] hold= 5,888 used= 1,440 count= 60 avg_used= 24 block_cnt= 7 chunk_cnt= 2 mod=[T]MemoryContex [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 7,859,136 used= 7,522,400 count= 304 avg_used= 24,744 mod=SUMMARY [2024-04-22 16:23:05.127174] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=56] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 31,457,280 used= 30,965,760 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 30,965,760 used= 30,908,160 count= 60 avg_used= 515,136 block_cnt= 60 chunk_cnt= 15 mod=CoStack [MEMORY] hold= 30,965,760 used= 30,908,160 count= 60 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:05.126482] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:3, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1713773681580810, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=9, offset=1469}, ls_meta_entry:[5](ver=0,mode=0,seq=4), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:53, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:222647231, recv_hp_rpc_cnt:1, recv_np_rpc_cnt:913, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:1369600, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=15405 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:2, queue_size:0, recv_req_cnt:2127, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773684733335}{group_id:10, queue_size:0, recv_req_cnt:1333, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773689875802}{group_id:5, queue_size:0, recv_req_cnt:801, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773687232900}{group_id:19, queue_size:0, recv_req_cnt:40, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1713774180915685}{group_id:9, queue_size:0, recv_req_cnt:12012, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773677273264}{group_id:1, queue_size:0, recv_req_cnt:5708, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1713774144054964}, rpc_stat_info: pcode=0x288:cnt=160 pcode=0x14a:cnt=129 pcode=0x502:cnt=68 pcode=0x1501:cnt=22 pcode=0x4a9:cnt=20, token_change_ts:1713773687300994, tenant_role:1}) [2024-04-22 16:23:05.127559] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=64] [MEMORY] tenant: 509, limit: 1,073,741,824 hold: 25,747,456 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 6,873,088 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 18,874,368 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:05.127673] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=33] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 6,873,088 used= 5,994,736 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,580,480 used= 2,410,520 count= 35 avg_used= 68,872 block_cnt= 35 chunk_cnt= 2 mod=[T]ObSessionDIB [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 16,800 used= 14,560 count= 35 avg_used= 416 block_cnt= 4 chunk_cnt= 1 mod=OMT_Worker [MEMORY] hold= 9,648 used= 4,480 count= 70 avg_used= 64 block_cnt= 4 chunk_cnt= 1 mod=Coro [MEMORY] hold= 3,360 used= 840 count= 35 avg_used= 24 block_cnt= 4 chunk_cnt= 1 mod=[T]MemoryContex [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 5,994,736 used= 5,786,400 count= 179 avg_used= 32,326 mod=SUMMARY [2024-04-22 16:23:05.127866] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=59] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 18,874,368 used= 18,063,360 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 18,063,360 used= 18,029,760 count= 35 avg_used= 515,136 block_cnt= 35 chunk_cnt= 9 mod=CoStack [MEMORY] hold= 18,063,360 used= 18,029,760 count= 35 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:05.128223] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=47] [MEMORY] tenant: 1, limit: 4,294,967,296 hold: 651,026,432 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 455,970,816 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 130,023,424 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 2,097,152 limit= 858,993,440 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 18,894,848 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33,554,432 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:05.127570] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1084] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:5, max_cpu:5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713773672949917, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"5.000000000000000000e+00", unit_max_cpu:"5.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:52, max_worker_cnt:150, stopped:0, worker_us:86586748, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2300684, workers:52, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1713773687322359, tenant_role:0}) [2024-04-22 16:23:05.128487] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=913] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:2.5, max_cpu:2.5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713773672930258, is_removed:false}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"2.500000000000000000e+00", unit_max_cpu:"2.500000000000000000e+00", total_worker_cnt:35, min_worker_cnt:27, max_worker_cnt:150, stopped:0, worker_us:57164117, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:1149773, workers:27, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1713773687342553, tenant_role:0}) [2024-04-22 16:23:05.129312] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.129353] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=38][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.129399] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=31] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd152a030, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240292, assoc_session_id:3221240292, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185129109, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185125782, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240292, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd152a030}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.129889] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.129612] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=74] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 455,970,816 used= 379,141,328 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 30 [MEMORY] hold= 157,024,256 used= 156,961,472 count= 39 avg_used= 4,024,653 block_cnt= 39 chunk_cnt= 39 mod=MysqlRequesReco [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 23,678,976 used= 23,614,912 count= 23 avg_used= 1,026,735 block_cnt= 23 chunk_cnt= 12 mod=IoControl [MEMORY] hold= 16,662,528 used= 15,565,072 count= 226 avg_used= 68,872 block_cnt= 226 chunk_cnt= 32 mod=[T]ObSessionDIB [MEMORY] hold= 14,483,248 used= 14,360,745 count= 93 avg_used= 154,416 block_cnt= 33 chunk_cnt= 7 mod=OmtTenant [MEMORY] hold= 8,986,624 used= 8,945,664 count= 2 avg_used= 4,472,832 block_cnt= 2 chunk_cnt= 2 mod=ServerObjecPool [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,257,536 used= 8,240,064 count= 2 avg_used= 4,120,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 5,234,688 used= 4,653,056 count= 71 avg_used= 65,536 block_cnt= 71 chunk_cnt= 9 mod=[T]char [MEMORY] hold= 4,943,872 used= 4,915,456 count= 2 avg_used= 2,457,728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4,571,136 used= 4,525,056 count= 6 avg_used= 754,176 block_cnt= 6 chunk_cnt= 5 mod=ResourceGroup [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 4,214,784 used= 4,194,336 count= 1 avg_used= 4,194,336 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 4 mod=SqlDtlQueue [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,864 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,166,784 used= 2,142,208 count= 2 avg_used= 1,071,104 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 2,080,768 used= 2,079,744 count= 1 avg_used= 2,079,744 block_cnt= 1 chunk_cnt= 1 mod=LogReplayTask [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 2 mod=DagMap [MEMORY] hold= 1,777,664 used= 1,770,696 count= 1 avg_used= 1,770,696 block_cnt= 1 chunk_cnt= 1 mod=TableProc [MEMORY] hold= 1,596,912 used= 1,588,632 count= 3 avg_used= 529,544 block_cnt= 3 chunk_cnt= 2 mod=TabletMap [MEMORY] hold= 1,584,288 used= 1,575,992 count= 3 avg_used= 525,330 block_cnt= 2 chunk_cnt= 1 mod=PartitTableTask [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1,269,184 used= 1,263,904 count= 6 avg_used= 210,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 786,560 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 3 mod=LogAlloc [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 655,360 used= 650,752 count= 1 avg_used= 650,752 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 446,976 used= 440,768 count= 97 avg_used= 4,544 block_cnt= 97 chunk_cnt= 17 mod=[T]ObTraceEvent [MEMORY] hold= 442,368 used= 438,432 count= 3 avg_used= 146,144 block_cnt= 3 chunk_cnt= 2 mod=SQLSessionInfo [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 403,648 used= 378,880 count= 258 avg_used= 1,468 block_cnt= 53 chunk_cnt= 3 mod=LSMap [MEMORY] hold= 385,024 used= 377,120 count= 1 avg_used= 377,120 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 365,040 used= 362,160 count= 45 avg_used= 8,048 block_cnt= 45 chunk_cnt= 11 mod=[T]ObPerfEventR [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 212,992 used= 212,160 count= 1 avg_used= 212,160 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 196,608 used= 196,224 count= 3 avg_used= 65,408 block_cnt= 3 chunk_cnt= 3 mod=StorMetaCacheIO [MEMORY] hold= 152,000 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 9 mod=LSIter [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 78,408 count= 9 avg_used= 8,712 block_cnt= 9 chunk_cnt= 6 mod=[T]ObDSActionAr [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 100,008 count= 1 avg_used= 100,008 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 98,304 used= 96,032 count= 1 avg_used= 96,032 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 97,408 count= 1 avg_used= 97,408 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 94,240 used= 90,112 count= 64 avg_used= 1,408 block_cnt= 17 chunk_cnt= 5 mod=TxCtxMgr [MEMORY] hold= 81,920 used= 65,536 count= 2 avg_used= 32,768 block_cnt= 2 chunk_cnt= 1 mod=CsDictPl [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 72,240 used= 1,024 count= 896 avg_used= 1 block_cnt= 10 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 64,624 used= 31,376 count= 455 avg_used= 68 block_cnt= 125 chunk_cnt= 32 mod=Coro [MEMORY] hold= 56,448 used= 55,584 count= 12 avg_used= 4,632 block_cnt= 12 chunk_cnt= 5 mod=WorkerMap [MEMORY] hold= 49,152 used= 48,576 count= 1 avg_used= 48,576 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 48,832 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=DictPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=RawPl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=ConstPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=CsIntPl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=CsStrPl [MEMORY] hold= 40,000 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 36,064 used= 32,928 count= 49 avg_used= 672 block_cnt= 20 chunk_cnt= 8 mod=[T]ObWarningBuf [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_ClientTask [MEMORY] hold= 32,768 used= 17,408 count= 2 avg_used= 8,704 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTTP [MEMORY] hold= 32,768 used= 25,600 count= 1 avg_used= 25,600 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 32,576 used= 26,079 count= 2 avg_used= 13,039 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 25,440 used= 22,048 count= 53 avg_used= 416 block_cnt= 14 chunk_cnt= 6 mod=OMT_Worker [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=IntDiffPl [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=RlePl [MEMORY] hold= 22,160 used= 5,424 count= 226 avg_used= 24 block_cnt= 91 chunk_cnt= 26 mod=[T]MemoryContex [MEMORY] hold= 17,472 used= 16,128 count= 21 avg_used= 768 block_cnt= 18 chunk_cnt= 6 mod=TGTimer [MEMORY] hold= 16,800 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 3 chunk_cnt= 3 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 13,696 count= 1 avg_used= 13,696 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 12,288 count= 1 avg_used= 12,288 block_cnt= 1 chunk_cnt= 1 mod=DiskIteratorSto [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=StrDiffPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=HexStrPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ColEqualPl [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxIMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_TableCtxMgr [MEMORY] hold= 15,744 used= 15,616 count= 2 avg_used= 7,808 block_cnt= 2 chunk_cnt= 2 mod=HashNodeLCSta [MEMORY] hold= 8,320 used= 8,192 count= 2 avg_used= 4,096 block_cnt= 2 chunk_cnt= 2 mod=LinearHashMapCn [MEMORY] hold= 8,320 used= 8,192 count= 2 avg_used= 4,096 block_cnt= 2 chunk_cnt= 2 mod=LinearHashMapDi [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTableMgr [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=TabletIDSetNode [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePlanCac [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoNode [MEMORY] hold= 7,856 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,856 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletNode [MEMORY] hold= 5,184 used= 5,120 count= 1 avg_used= 5,120 block_cnt= 1 chunk_cnt= 1 mod=CtxBlockPl [MEMORY] hold= 4,800 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=CaLsInfoMap [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,160 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=CsCtxBlockPl [MEMORY] hold= 4,160 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 3,072 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,336 used= 1,728 count= 8 avg_used= 216 block_cnt= 4 chunk_cnt= 3 mod=LogIOTask [MEMORY] hold= 1,984 used= 1,920 count= 1 avg_used= 1,920 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 1,872 used= 1,800 count= 1 avg_used= 1,800 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,872 used= 1,296 count= 9 avg_used= 144 block_cnt= 5 chunk_cnt= 4 mod=ObTxLSLogCb [MEMORY] hold= 1,664 used= 1,600 count= 1 avg_used= 1,600 block_cnt= 1 chunk_cnt= 1 mod=LogPartFetCtxPo [MEMORY] hold= 1,616 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,488 used= 1,352 count= 2 avg_used= 676 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=LockMemtableMgr [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,184 used= 1,056 count= 2 avg_used= 528 block_cnt= 1 chunk_cnt= 1 mod=PoolArenaArray [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,120 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 848 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=TabShellTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=DiskUsageTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=FlushTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=CheckPointTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=TLD_TIMER [MEMORY] hold= 720 used= 648 count= 1 avg_used= 648 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 624 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 544 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 480 used= 120 count= 5 avg_used= 24 block_cnt= 3 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 336 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 192 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 192 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 176 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 176 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 160 used= 16 count= 2 avg_used= 8 block_cnt= 2 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 128 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 112 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 80 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 379,141,328 used= 372,914,203 count= 28,353 avg_used= 13,152 mod=SUMMARY [2024-04-22 16:23:05.129955] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:39, local_retry_times:39, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.130045] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=81][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.130064] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.130074] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=9][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.130132] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:05.130138] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=382] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 794,624 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 794,624 used= 788,032 count= 1 avg_used= 788,032 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 794,624 used= 788,032 count= 1 avg_used= 788,032 mod=SUMMARY [2024-04-22 16:23:05.130214] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=32] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 2,097,152 used= 1,286,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,232,192 used= 1,115,344 count= 43 avg_used= 25,938 block_cnt= 43 chunk_cnt= 1 mod=SqlPhyPlan [MEMORY] hold= 32,000 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 1 mod=SqlPlanCache [MEMORY] hold= 20,608 used= 19,840 count= 10 avg_used= 1,984 block_cnt= 3 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 1,728 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 1,286,528 used= 1,168,528 count= 59 avg_used= 19,805 mod=SUMMARY [2024-04-22 16:23:05.130338] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=39] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,848 used= 59,018 count= 263 avg_used= 224 block_cnt= 20 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,944 used= 59,050 count= 264 avg_used= 223 mod=SUMMARY [2024-04-22 16:23:05.130424] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=49] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 130,023,424 used= 126,443,520 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 126,443,520 used= 126,208,320 count= 245 avg_used= 515,136 block_cnt= 245 chunk_cnt= 62 mod=CoStack [MEMORY] hold= 126,443,520 used= 126,208,320 count= 245 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:05.130553] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=69] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 2,097,152 used= 466,944 limit= 858,993,440 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 401,408 used= 400,064 count= 2 avg_used= 200,032 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=N_TabletPool [MEMORY] hold= 466,944 used= 465,472 count= 3 avg_used= 155,157 mod=SUMMARY [2024-04-22 16:23:05.130622] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=34] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 65,536 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=LobAllocator [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 mod=SUMMARY [2024-04-22 16:23:05.130713] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=64] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 2,097,152 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2024-04-22 16:23:05.130796] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=16] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 18,894,848 used= 17,870,464 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17,862,464 used= 17,833,792 count= 3 avg_used= 5,944,597 block_cnt= 3 chunk_cnt= 2 mod=MEMTABLE_ARENA [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 17,870,464 used= 17,841,728 count= 4 avg_used= 4,460,432 mod=SUMMARY [2024-04-22 16:23:05.130925] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=32] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33,554,432 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 33,554,432 free_size= 33,554,432 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2024-04-22 16:23:05.131090] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:122) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=33] ====== tenants memory info ====== === TENANTS MEMORY INFO === divisive_memory_used= 86,245,376 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 579,035,136 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 40,427,520 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 25,747,456 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,713,774,185,128,165 active_memstore_used= 0 total_memstore_used= 0 total_memstore_hold= 0 memstore_freeze_trigger_limit= 343,597,360 memstore_limit= 1,717,986,880 mem_tenant_limit= 4,294,967,296 mem_tenant_hold= 651,026,432 max_mem_memstore_can_get_now= 1,821,970,400 memstore_alloc_pos= 0 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2024-04-22 16:23:05.131102] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=107][errcode=0] server is initiating(server_id=0, local_seq=14821, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.131187] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=34][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1069826) [2024-04-22 16:23:05.131239] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=48][errcode=-8001] [2024-04-22 16:23:05.131297] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=20] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.131572] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:162) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=68] [CHUNK_MGR] free=13 pushes=61561 pops=61548 limit= 26,843,545,600 hold= 1,329,823,744 total_hold= 1,421,869,056 used= 1,298,333,696 freelist_hold= 31,490,048 large_freelist_hold= 4,227,072 maps= 434 unmaps= 12 large_maps= 34 large_unmaps= 5 huge_maps= 32 huge_unmaps= 3 memalign=0 resident_size= 1,769,279,488 virtual_memory_used= 2,297,921,536 [2024-04-22 16:23:05.131696] INFO print (ob_malloc_time_monitor.cpp:39) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=64] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 0, 10): delta_total_cost_time= 1070889, delta_count= 7768939, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10, 100): delta_total_cost_time= 13382, delta_count= 848, avg_cost_time= 15 [MALLOC_TIME_MONITOR] [ 100, 1000): delta_total_cost_time= 355, delta_count= 3, avg_cost_time= 118 [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2024-04-22 16:23:05.131853] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=37][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1069826) [2024-04-22 16:23:05.131913] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=54][errcode=-8001] [2024-04-22 16:23:05.131964] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.132674] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=25][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1069826) [2024-04-22 16:23:05.132718] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=39][errcode=-8001] [2024-04-22 16:23:05.132767] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.133133] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.133195] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=58][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.133294] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1069826) [2024-04-22 16:23:05.133235] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd152bf50, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240293, assoc_session_id:3221240293, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185132469, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185132469, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240293, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd152bf50}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.133349] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=50][errcode=-8001] [2024-04-22 16:23:05.133387] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:05.133600] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.133583] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=47] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185133488), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.133653] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=63][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:05.133677] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=23][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:05.133696] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:05.133736] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=36][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:05.133804] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=65][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=40, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:05.133841] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=23] will sleep(sleep_us=40000, remain_us=1066533, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.135654] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC99-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.135725] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC99-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.141007] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.141104] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185115836, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185108190, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.141509] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=29685633, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.151101] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=50][errcode=0] server is initiating(server_id=0, local_seq=14822, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.152757] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.152802] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=43][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.152830] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=17] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd152fd90, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240294, assoc_session_id:3221240294, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185152359, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185150134, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240294, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd152fd90}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.153111] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.156105] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=43][errcode=0] server is initiating(server_id=0, local_seq=14823, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.157256] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=58] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.157352] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185103807, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185104381, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.157795] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=53000, remain_us=583345, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.157904] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.157952] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=46][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.157984] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=20] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1531cb0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240295, assoc_session_id:3221240295, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185156835, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185155714, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240295, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1531cb0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.158306] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.166725] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.166824] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185140161, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185132470, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.167153] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=26000, remain_us=29659989, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.167549] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=70] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:05.168070] WDIAG [SHARE] get_tenant_primary_zone_score (ob_primary_zone_util.cpp:654) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC75-0-0] [lt=14][errcode=-4016] zone list count unexpected(ret=-4016) [2024-04-22 16:23:05.168120] WDIAG [SHARE] get_tenant_primary_zone_array (ob_primary_zone_util.cpp:613) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC75-0-0] [lt=45][errcode=-4016] failed to get tenant primary zone score(ret=-4016, ret="OB_ERR_UNEXPECTED", tenant_schema={tenant_id:1, schema_version:1, tenant_name:"sys", zone_list:[cnt:0], primary_zone:"", charset_type:2, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:"", zone_replica_attr_array:[cnt:0], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:-1, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2024-04-22 16:23:05.168404] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:588) [10807][T1_TntSharedTim][T1][Y0-0000000000000000-0-0] [lt=52] dump dtl interm result cost(us)(dump_cost=4412, ret=0, interm count=0, dump count=50) [2024-04-22 16:23:05.168682] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:05.168744] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:05.168785] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774185168664) [2024-04-22 16:23:05.168847] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774184968622, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:05.172711] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:603) [10807][T1_TntSharedTim][T1][Y0-0000000000000000-0-0] [lt=88] clear dtl interm result cost(us)(clear_cost=4215, ret=0, gc_.expire_keys_.count()=0, dump count=50, interm count=0, clean count=0) [2024-04-22 16:23:05.174021] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=37][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:40, local_retry_times:40, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.174095] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=64][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.174123] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=25][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.174135] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.174160] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:05.175272] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1025863) [2024-04-22 16:23:05.175324] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=46][errcode=-8001] [2024-04-22 16:23:05.175371] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.176090] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=24][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1025863) [2024-04-22 16:23:05.176135] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=40][errcode=-8001] [2024-04-22 16:23:05.176187] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=30] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.176955] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1025863) [2024-04-22 16:23:05.177022] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=60][errcode=-8001] [2024-04-22 16:23:05.177088] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=27] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.177404] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=48][errcode=0] server is initiating(server_id=0, local_seq=14824, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.177811] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=46][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=1025863) [2024-04-22 16:23:05.177878] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=61][errcode=-8001] [2024-04-22 16:23:05.177918] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=24][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:05.178128] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=68] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185177991), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.178185] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=51][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:05.178218] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=29][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:05.178240] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:05.178260] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:05.178335] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=72][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=41, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:05.178386] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=35] will sleep(sleep_us=41000, remain_us=1021987, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.178510] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED34-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.178582] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED34-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:05.179418] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.179505] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=84][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.179556] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=37] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1533bd0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240296, assoc_session_id:3221240296, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185179011, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185176821, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240296, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1533bd0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.179860] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.180377] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9A-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.180489] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9A-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.181975] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=48][errcode=0] server is initiating(server_id=0, local_seq=14825, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.183154] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=51] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:05.183808] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.183869] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=58][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.183905] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=24] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1535af0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240297, assoc_session_id:3221240297, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185183560, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185181245, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240297, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1535af0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.184212] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.193387] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.193517] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185166947, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185155714, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.193805] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] get number(ret=0, service_type_=0, range=1, base_id=1713774185193802652, start_id=1713774187441983331, end_id=1713774187441983332) [2024-04-22 16:23:05.193853] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=43][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774185193802652, last_request_ts=1713774185092866045, time_delta=100936607, last_gts=1713774187431612540, gts=1713774187441983332, gts_delta=10370791, compensation_value=10093660) [2024-04-22 16:23:05.193957] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=27000, remain_us=29633185, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.194069] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [10492][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=91] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService13ObRestartTaskE) [2024-04-22 16:23:05.196479] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [10492][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=53] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-04-22 16:23:05.198590] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBEA-0-0] [lt=52][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.198659] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBEA-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:05.198697] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10493][RSAsyncTask0][T1][YB42AC175881-000616AB09FFEBEA-0-0] [lt=35][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.208801] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=47][errcode=0] server is initiating(server_id=0, local_seq=14826, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.210756] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.210815] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=57][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.210854] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1537a10, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240298, assoc_session_id:3221240298, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185210070, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185207870, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240298, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1537a10}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.211052] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=76] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.211148] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185156835, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185155714, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.211277] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.211599] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=54000, remain_us=529541, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.219539] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=30][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:41, local_retry_times:41, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.219608] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=60][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.219646] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=35][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.219676] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=27][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.219743] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=29][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:05.220701] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=39][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=980316) [2024-04-22 16:23:05.220806] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=62][errcode=-8001] [2024-04-22 16:23:05.220865] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=30] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.221191] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=59] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.221357] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185193502, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185181245, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.221763] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=47][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=980316) [2024-04-22 16:23:05.221811] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=43][errcode=-8001] [2024-04-22 16:23:05.221797] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=28000, remain_us=29605345, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.221842] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=11] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.222471] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=31][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=980316) [2024-04-22 16:23:05.222514] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=39][errcode=-8001] [2024-04-22 16:23:05.222557] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.223173] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=30][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=980316) [2024-04-22 16:23:05.223230] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=53][errcode=-8001] [2024-04-22 16:23:05.223257] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:05.223380] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=47] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185223308), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.223423] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=38][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:05.223494] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=68][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:05.223524] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=27][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:05.223554] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:05.223589] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=32][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=42, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:05.223643] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=38] will sleep(sleep_us=42000, remain_us=976731, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.225338] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9B-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.225427] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9B-0-0] [lt=70] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.225513] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9B-0-0] [lt=81][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.228892] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=33][errcode=0] server is initiating(server_id=0, local_seq=14827, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.230866] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.230957] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=88][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.231007] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=32] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1539930, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240299, assoc_session_id:3221240299, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185230059, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185230058, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240299, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1539930}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.231379] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.234979] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:05.235064] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05C-0-0] [lt=74][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:05.235114] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05C-0-0] [lt=46][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:05.235160] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05C-0-0] [lt=43][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:05.235195] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05C-0-0] [lt=34][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:05.236124] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05E-0-0] [lt=303][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:05.236210] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05E-0-0] [lt=83][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:05.236256] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05E-0-0] [lt=41][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:05.236316] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05E-0-0] [lt=57][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:05.236761] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05E-0-0] [lt=311][errcode=0] compare between invalid priority [2024-04-22 16:23:05.236743] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=105][errcode=0] server is initiating(server_id=0, local_seq=14828, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.236858] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05D-0-0] [lt=246][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:05.236920] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05D-0-0] [lt=58][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:05.236966] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05D-0-0] [lt=44][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:05.237019] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05D-0-0] [lt=50][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:05.237386] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10918][T1_L0_G2][T1][YB42AC175881-000616AB08AFF05D-0-0] [lt=255][errcode=0] compare between invalid priority [2024-04-22 16:23:05.238562] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.238614] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=49][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.238651] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=24] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd153b850, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240300, assoc_session_id:3221240300, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185237792, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185235539, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240300, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153b850}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.238951] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.244145] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7A-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.244176] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7A-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:05.244191] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7A-0-0] [lt=13][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.250006] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.250106] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185221125, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185221124, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.250546] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=29000, remain_us=29576596, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.255309] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=47][errcode=0] server is initiating(server_id=0, local_seq=14829, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.256983] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1806) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=56][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1713774185198760, current_time=1713774185256881) [2024-04-22 16:23:05.257052] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1822) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=52][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1713774185198760, now=[mts=1713774185256981], now0=[mts=1713774185256981], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-04-22 16:23:05.257092] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:657) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=36][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f3cd153f690, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240301, assoc_session_id:3221240301, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185256881, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185254564, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:05.257196] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:736) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=91][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x7f3cd1157048, id:3221240301, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153f690}) [2024-04-22 16:23:05.257257] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:566) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=50][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=3221240301, *tx_desc={this:0x7f3cd153f690, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240301, assoc_session_id:3221240301, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185256881, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185254564, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:05.257363] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:636) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=102] start stmt(ret=-4012, auto_commit=true, session_id=3221240301, snapshot={this:0x7f3cbde09a28, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, savepoint=0, tx_desc={this:0x7f3cd153f690, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240301, assoc_session_id:3221240301, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185256881, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185254564, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=2, stmt_type=1, has_for_update=false, query_start_time=1713774155268789, use_das=false, nested_level=0, session={this:0x7f3cd1157048, id:3221240301, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd153f690}, plan=0x7f3cbde0a3a8, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-04-22 16:23:05.257579] WDIAG [SQL.EXE] auto_start_phy_trans (ob_remote_executor_processor.cpp:236) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=212][errcode=-4012] fail to start stmt(ret=-4012) [2024-04-22 16:23:05.257603] WDIAG [SQL.EXE] execute_remote_plan (ob_remote_executor_processor.cpp:450) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=21][errcode=-4012] fail to begin trans(ret=-4012) [2024-04-22 16:23:05.257618] WDIAG [SQL.EXE] process (ob_remote_executor_processor.cpp:990) [10882][T1_L5_G0][T1][YB42AC175882-000616AB0B2C2FB7-0-0] [lt=12][errcode=-4012] execute remote plan failed(ret=-4012, local_tenant_schema_version=1, exec_ctx_.get_task_exec_ctx().get_query_tenant_begin_schema_version()=1) [2024-04-22 16:23:05.265552] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=13][errcode=0] server is initiating(server_id=0, local_seq=14830, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.265820] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=36][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:42, local_retry_times:42, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.265855] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=45] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.265891] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=61][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.265919] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=25][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.265933] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.265958] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=12][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:05.265941] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185211175, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185207870, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.266308] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=55000, remain_us=474832, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.267057] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=934122) [2024-04-22 16:23:05.267118] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=55][errcode=-8001] [2024-04-22 16:23:05.267164] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=18] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.267494] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.267556] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=58][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.267598] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15415b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240302, assoc_session_id:3221240302, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185266866, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185266865, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240302, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15415b0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.268055] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.268290] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=42][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=934122) [2024-04-22 16:23:05.268355] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=59][errcode=-8001] [2024-04-22 16:23:05.268371] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18] ====== tenant freeze timer task ====== [2024-04-22 16:23:05.268421] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=35] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.268731] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:778) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=7, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:05.268800] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:358) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-04-22 16:23:05.268830] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] start TenantWeakReadClusterService(tenant_id=1) [2024-04-22 16:23:05.269096] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=79][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=934122) [2024-04-22 16:23:05.269140] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=39][errcode=-8001] [2024-04-22 16:23:05.269181] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.269502] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:05.269788] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=34][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=934122) [2024-04-22 16:23:05.269832] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=40][errcode=-8001] [2024-04-22 16:23:05.269860] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:05.269968] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=34] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185269897), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.270037] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=62][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:05.270061] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:05.270080] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:05.270097] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:05.270171] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=71][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=43, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:05.270216] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=28] will sleep(sleep_us=43000, remain_us=930158, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.270776] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECF2-0-0] [lt=16][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.270860] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10842][T1_TenantWeakRe][T1][YB42AC175881-000616AB0A4FECF2-0-0] [lt=65] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-04-22 16:23:05.271049] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:449) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1713774185271047, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=2259, wlock_time=59, check_leader_time=3, query_version_time=0, persist_version_time=0) [2024-04-22 16:23:05.271152] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=7, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:05.271971] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9C-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.272037] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9C-0-0] [lt=49] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.279753] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.279850] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185250030, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185235539, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.280212] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=30000, remain_us=29546930, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.282888] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.282955] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15434d0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240303, assoc_session_id:3221240303, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185282607, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185280345, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240303, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15434d0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.283319] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.288164] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=79] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:05.297567] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=2] get number(ret=0, service_type_=0, range=1, base_id=1713774185297563535, start_id=1713774187452077005, end_id=1713774187452077006) [2024-04-22 16:23:05.297721] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.297805] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd15453f0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240304, assoc_session_id:3221240304, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185297337, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185295083, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240304, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd15453f0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.298143] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.303231] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=82][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:5, tid:10479}]) [2024-04-22 16:23:05.310489] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.310635] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185279113, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185269056, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.311098] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=31000, remain_us=29516044, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.313369] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=41][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:43, local_retry_times:43, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.313502] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=122][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.313532] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=26][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.313551] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.313599] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:05.314864] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=886442) [2024-04-22 16:23:05.314935] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=64][errcode=-8001] [2024-04-22 16:23:05.314989] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=22] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.315867] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=37][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=886442) [2024-04-22 16:23:05.315918] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=46][errcode=-8001] [2024-04-22 16:23:05.315963] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.316807] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=886442) [2024-04-22 16:23:05.316902] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=88][errcode=-8001] [2024-04-22 16:23:05.316949] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.317714] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] execute rpc fail(addr="172.23.88.129:2882", pcode=648, ret=-8001, timeout=886442) [2024-04-22 16:23:05.317789] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=69][errcode=-8001] [2024-04-22 16:23:05.317829] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1241) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=24][errcode=-8001] set config rpc failed(ret=-8001, rpc_arg={items:[{name:"rootservice_list", value:"172.23.88.129:2882:2881;172.23.88.131:2882:2881", comment:"", zone:"", server:"0.0.0.0:0", tenant_name:"", exec_tenant_id:1, tenant_ids:[], want_to_set_tenant_config:false}], is_inner:false}) [2024-04-22 16:23:05.317954] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=45] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185317878), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.317998] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=39][errcode=-8001] execute cmd failed(ret=-8001) [2024-04-22 16:23:05.318021] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21][errcode=-8001] execute plan failed(ret=-8001) [2024-04-22 16:23:05.318036] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=13][errcode=-8001] open result set failed(ret=-8001) [2024-04-22 16:23:05.318048] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-8001] result set open failed(ret=-8001, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:05.318066] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=17][errcode=-8001] execute failed(ret=-8001, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=44, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:05.318118] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=36] will sleep(sleep_us=44000, remain_us=882255, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.320004] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9D-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.320076] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9D-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.321611] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=57] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.321860] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:05.321774] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185265761, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185264652, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.322166] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=56000, remain_us=418975, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.326737] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=69][errcode=0] server is initiating(server_id=0, local_seq=14833, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.328770] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.328821] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=48][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.328879] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=40] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1547310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240305, assoc_session_id:3221240305, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185328565, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185326356, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240305, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1547310}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.329261] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.333940] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=50][errcode=0] server is initiating(server_id=0, local_seq=14834, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.335987] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.336034] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=45][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.336072] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1549230, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240306, assoc_session_id:3221240306, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185335378, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185335377, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240306, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1549230}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.336368] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.342422] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.342584] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185310754, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185295083, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.342846] INFO [SQL.ENG] reset_leader_statistics (ob_px_tenant_target_monitor.cpp:382) [10963][T1_L0_G0][T1][YB42AC175882-000616AB096C3059-0-0] [lt=67] reset leader statistics(tenant_id_=1, ret=0, version_=24924325078906, GCTX.server_id_=0) [2024-04-22 16:23:05.342907] INFO [SQL.ENG] process (ob_px_rpc_processor.cpp:613) [10963][T1_L0_G0][T1][YB42AC175882-000616AB096C3059-0-0] [lt=55] need refresh all(tenant_id=1, arg_.need_refresh_all_=true, follower_version=18446744073709551615, prev_leader_server_id=65535, leader_server_id=0) [2024-04-22 16:23:05.342951] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=32000, remain_us=29484191, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.359180] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=49][errcode=0] server is initiating(server_id=0, local_seq=14835, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.361186] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.361240] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=51][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.361271] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=19] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154b150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240307, assoc_session_id:3221240307, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185361086, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185358836, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240307, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154b150}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.361721] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.362287] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=37][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:44, local_retry_times:44, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:05.362362] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=65][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.362391] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=25][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:05.362409] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=16][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:05.362481] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=19][errcode=-8001] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.363621] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.364617] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.365426] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.366267] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185366194), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.366343] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] will sleep(sleep_us=45000, remain_us=834031, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.368283] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9E-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.368354] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9E-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.368840] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=98][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:05.368934] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774185368826) [2024-04-22 16:23:05.368976] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774185168872, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:05.375245] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=72] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.375359] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185342167, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185342166, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.375814] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=33000, remain_us=29451328, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.378392] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=49] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.378530] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185320820, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185317474, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.378922] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=57000, remain_us=362219, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.388231] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10953][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=55][errcode=0] server is initiating(server_id=0, local_seq=14836, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.390252] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10953][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=93][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.390351] WDIAG push_back (ob_fixed_array.h:71) [10953][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.390394] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10953][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=41][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.390430] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10953][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=20] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd23347b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240308, assoc_session_id:3221240308, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185389346, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185387090, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc86f048, id:3221240308, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd23347b0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.390812] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10953][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.392827] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=88][errcode=0] server is initiating(server_id=0, local_seq=14837, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.394990] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.395039] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=47][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.395131] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=32] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154d070, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240309, assoc_session_id:3221240309, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185393871, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185391606, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240309, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154d070}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.395551] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.409029] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=48] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.409132] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185374583, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185365495, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.409408] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] get number(ret=0, service_type_=0, range=1, base_id=1713774185409405609, start_id=1713774187462453107, end_id=1713774187462453108) [2024-04-22 16:23:05.409501] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=89][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774185409405609, last_request_ts=1713774185297563535, time_delta=111842074, last_gts=1713774187452077006, gts=1713774187462453108, gts_delta=10376101, compensation_value=11184207) [2024-04-22 16:23:05.409619] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=34000, remain_us=29417523, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.413162] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.414205] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.414959] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.415778] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185415667), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.415878] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=46000, remain_us=784495, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.417727] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9F-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.417834] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFEC9F-0-0] [lt=88] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.427308] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=54][errcode=0] server is initiating(server_id=0, local_seq=14838, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.429102] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.429173] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=69][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.429212] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd154ef90, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240310, assoc_session_id:3221240310, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185427944, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185426839, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240310, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd154ef90}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.429779] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=2] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.435791] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=91] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:05.436221] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=60] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.436325] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185377920, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185365495, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.436719] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=58000, remain_us=304421, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.443877] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=42] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.443826] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10887][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=13][errcode=0] server is initiating(server_id=0, local_seq=14839, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.444010] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185408747, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185391606, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.444391] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=35000, remain_us=29382751, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.445971] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10887][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=149][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.446072] WDIAG push_back (ob_fixed_array.h:71) [10887][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.446101] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10887][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=26][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.446133] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10887][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=16] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cc001d770, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240311, assoc_session_id:3221240311, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185444850, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185442589, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc880ce8, id:3221240311, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cc001d770}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.446588] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10887][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.460040] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=97] BlockGCTimerTask success(ret=0, cost_time_us=41, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-04-22 16:23:05.463501] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.464381] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.465049] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.465867] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185465739), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.465940] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=47000, remain_us=734433, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.467858] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA0-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.467926] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA0-0-0] [lt=49] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.468364] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=51][errcode=0] server is initiating(server_id=0, local_seq=14840, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.469710] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:05.470385] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.470472] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=85][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.470525] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=39] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1550eb0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240312, assoc_session_id:3221240312, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185469825, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185467586, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240312, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1550eb0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.470836] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.479654] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=94] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.479754] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185443732, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185442589, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.480168] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=36000, remain_us=29346975, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.494987] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=44] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.495066] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185435810, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185426839, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.495349] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=59000, remain_us=245791, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=59, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.500626] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=48][errcode=0] server is initiating(server_id=0, local_seq=14841, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.502515] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.502570] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=52][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.502609] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1552dd0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240313, assoc_session_id:3221240313, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185502273, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185500073, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240313, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1552dd0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.502966] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.504428] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=62][errcode=0] server is initiating(server_id=0, local_seq=14842, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.506339] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.506393] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=52][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.506473] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=33] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1554cf0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240314, assoc_session_id:3221240314, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185505588, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185503536, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240314, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1554cf0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.506790] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.514156] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.514757] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.515490] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.516228] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185516120), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.516307] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=48000, remain_us=684066, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.516405] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=94] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.516488] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185479808, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185467586, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.516638] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] get number(ret=0, service_type_=0, range=1, base_id=1713774185516636562, start_id=1713774187473637326, end_id=1713774187473637327) [2024-04-22 16:23:05.516661] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774185516636562, last_request_ts=1713774185409405609, time_delta=107230953, last_gts=1713774187462453108, gts=1713774187473637327, gts_delta=11184218, compensation_value=10723095) [2024-04-22 16:23:05.516723] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=37000, remain_us=29310419, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.518133] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA1-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.518224] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA1-0-0] [lt=65] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.541808] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10961][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=115][errcode=0] server is initiating(server_id=0, local_seq=14843, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.543785] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10961][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=70][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.543882] WDIAG push_back (ob_fixed_array.h:71) [10961][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.543904] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10961][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=22][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.543932] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10961][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=15] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd23d8dd0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240315, assoc_session_id:3221240315, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185543378, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185541050, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3ccc8a7048, id:3221240315, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd23d8dd0}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.544284] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10961][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.553897] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.554085] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185515862, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185515861, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.554589] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=38000, remain_us=29272554, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.554582] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=44] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.554681] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185494493, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185467586, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.555221] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=1] will sleep(sleep_us=60000, remain_us=185920, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=60, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.557892] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=49][errcode=0] server is initiating(server_id=0, local_seq=14844, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.559767] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.559827] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=57][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.559880] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=33] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1558b30, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240316, assoc_session_id:3221240316, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185559353, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185557030, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240316, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1558b30}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.560255] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.565780] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.566391] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.567026] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.568006] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185567782), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.568118] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] will sleep(sleep_us=49000, remain_us=632256, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.568742] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:249) [10820][T1_RFLWorker][T1][YB42AC175881-000616AB070FEBAD-0-0] [lt=50] ObRemoteFetchWorker is running(thread_index=0) [2024-04-22 16:23:05.569027] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:05.569102] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774185569013) [2024-04-22 16:23:05.569133] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774185369001, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:05.568837] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2644) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=116] [PALF_DUMP](palf_id=1, self="172.23.88.129:2882", [SlidingWindow]={palf_id:1, self:"172.23.88.129:2882", lsn_allocator:{max_log_id:2941, max_lsn:358576, max_scn:{val:1713774187421454783, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:358576}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:358454}, last_submit_end_lsn:{lsn:358576}, last_submit_log_id:2941, last_submit_log_pid:7, max_flushed_lsn:{lsn:358454}, max_flushed_end_lsn:{lsn:358576}, max_flushed_log_pid:7, committed_end_lsn:{lsn:358576}, last_slide_log_id:2941, last_slide_scn:{val:1713774187421454783, v:0}, last_slide_lsn:{lsn:358454}, last_slide_end_lsn:{lsn:358576}, last_slide_log_pid:7, last_slide_log_accum_checksum:1087531090, last_fetch_end_lsn:{lsn:42299499}, last_fetch_max_log_id:4972, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713774179963810, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f3cd2de4530}, [StateMgr]={this:0x7f3cd2de8330, self:"172.23.88.129:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, leader:"172.23.88.129:2882", leader_epoch:10, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:338140}, scan_disk_log_finished:true, last_check_start_id:2942, is_changing_config_with_arb:false, reconfirm_start_time_us:1713774181765332, palf_role_change_cb:0x7f3cd2deebf0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.23.88.129:2882", alive_paxos_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_end_lsn:{lsn:356581}, prev_mode_pid:7}, reconfig_barrier:{prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_end_lsn:{lsn:356581}, prev_mode_pid:7}, persistent_config_version:{proposal_id:7, config_seq:8}, ms_ack_list:0, resend_config_version:{proposal_id:7, config_seq:8}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:1713774184767831, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1713774177861681, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1713774177861681, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f3cd2de5c70}, [ModeMgr]={palf_id:1, self:"172.23.88.129:2882", applied_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1713774183967179, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:7}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:102400}, readable_log_tail:{lsn:102400}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67002368, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}, log_storage:{log_tail:{lsn:358576}, readable_log_tail:{lsn:358576}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/log", dir_fd:235, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66746192, block_header_serialize_buf_:0x7f3cd2dec138, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1713774181764894, this:0x7f3cd2de8a70}, [Reconfirm]={palf_id:1, self:"172.23.88.129:2882", state:"FINISHED", new_proposal_id:7, prepare_log_ack_list:1"172.23.88.131:2882";, curr_paxos_follower_list:2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.23.88.129:2882", majority_max_accept_pid:6, majority_max_lsn:{lsn:356581}, saved_end_lsn:{lsn:356581}, last_submit_prepare_req_time_us:1713774181765723, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f3cd2de84b0}) [2024-04-22 16:23:05.569636] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=783] [PALF_DUMP](palf_id=1, self="172.23.88.129:2882", ack_info_list=[{member:{server:"172.23.88.129:2882", timestamp:1, flag:0}, last_ack_time_us:1713774184972896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.130:2882", timestamp:1, flag:0}, last_ack_time_us:1713774182165759, last_flushed_end_lsn:{lsn:356581}}, {member:{server:"172.23.88.131:2882", timestamp:1, flag:0}, last_ack_time_us:1713774184975493, last_flushed_end_lsn:{lsn:358576}}]) [2024-04-22 16:23:05.569777] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4287) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=102] current log_task status(palf_id=1, self="172.23.88.129:2882", log_id=2942, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.23.88.129:2882", timestamp:1, flag:0}, last_ack_time_us:1713774184972896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.130:2882", timestamp:1, flag:0}, last_ack_time_us:1713774182165759, last_flushed_end_lsn:{lsn:356581}}, {member:{server:"172.23.88.131:2882", timestamp:1, flag:0}, last_ack_time_us:1713774184975493, last_flushed_end_lsn:{lsn:358576}}]) [2024-04-22 16:23:05.570145] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA2-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.570223] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA2-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.580223] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=50][errcode=0] server is initiating(server_id=0, local_seq=14845, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.582223] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.582274] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=49][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.582313] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd1556c10, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240317, assoc_session_id:3221240317, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185581883, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185580781, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1168ce8, id:3221240317, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd1556c10}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.582731] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.582785] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFBE7-0-0] [lt=45][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774185582700}) [2024-04-22 16:23:05.592895] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=80] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.592975] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185553690, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185550191, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.593343] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=39000, remain_us=29233799, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.605912] INFO keepalive_check (time_wheel.c:76) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd24d9358,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:05.615506] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=62] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.615611] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=2] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185554805, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185550191, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.616027] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=0] will sleep(sleep_us=61000, remain_us=125113, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=61, v.err_=-4725, timeout_timestamp=1713774185741139) [2024-04-22 16:23:05.616157] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10957][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=90][errcode=0] server is initiating(server_id=0, local_seq=14846, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.618179] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10957][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=41] get number(ret=0, service_type_=0, range=1, base_id=1713774185618175634, start_id=1713774187484360431, end_id=1713774187484360432) [2024-04-22 16:23:05.618250] WDIAG [STORAGE.TRANS] get_timestamp (ob_timestamp_service.cpp:102) [10957][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=66][errcode=0] the gts service advanced too slowly(ret=0, current_time=1713774185618175634, last_request_ts=1713774185516636562, time_delta=101539072, last_gts=1713774187473637327, gts=1713774187484360432, gts_delta=10723104, compensation_value=10153907) [2024-04-22 16:23:05.618362] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10957][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=38][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.618483] WDIAG push_back (ob_fixed_array.h:71) [10957][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.618503] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10957][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=19][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.618529] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10957][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=14] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cc56f9310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240318, assoc_session_id:3221240318, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185617497, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185617496, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd115fe98, id:3221240318, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cc56f9310}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.618636] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.618869] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10957][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.619270] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.619734] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=50][errcode=0] server is initiating(server_id=0, local_seq=14847, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.620226] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:05.621370] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774185621162), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -8001, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:05.621489] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=2] will sleep(sleep_us=50000, remain_us=578885, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-8001, timeout_timestamp=1713774186200372) [2024-04-22 16:23:05.621728] WDIAG push_back (ob_fixed_array.h:71) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=0][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.621818] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=88][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.621857] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=25] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd155aa50, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240319, assoc_session_id:3221240319, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185620913, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185620912, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd114e1f8, id:3221240319, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155aa50}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.622250] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.623575] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA3-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:05.623648] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA3-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:05.632616] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=75] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:05.632719] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185592152, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185591039, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:05.633075] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=29194067, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:05.636758] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:05.660167] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10891][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=3][errcode=0] server is initiating(server_id=0, local_seq=14848, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:05.662287] WDIAG [STORAGE] construct_store_ctx_other_variables_ (ob_access_service.cpp:452) [10891][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=73][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:05.662383] WDIAG push_back (ob_fixed_array.h:71) [10891][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1][errcode=-4006] array not init(ret=-4006, capacity=0) [2024-04-22 16:23:05.662405] WDIAG [SQL.ENG] fill_storage_feedback_info (ob_table_scan_op.cpp:1554) [10891][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=21][errcode=0] push back table_id-row_count failed but we won't stop execution(tmp_ret=-4006) [2024-04-22 16:23:05.662433] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10891][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=15] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cc5e1f690, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240320, assoc_session_id:3221240320, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185661353, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185659132, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd1157048, id:3221240320, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cc5e1f690}, exec_ctx.get_errcode()=0) [2024-04-22 16:23:05.662964] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10891][T1_L0_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:05.669161] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:310) [10850][T1_ArcFetcher][T1][YB42AC175881-000616AB076FEBAD-0-0] [lt=51] ObArchiveFetcher is running(thread_index=0) [2024-04-22 16:23:05.669212] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:05.669283] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774185669193) [2024-04-22 16:23:05.669347] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774185569154, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:12.190798] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:406) [10819][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=40] finish to resize log external storage handler(current_ts=1713774192190565, tenant_max_cpu=3, valid_ls_count=0, other_ls_count=0, new_concurrency=0) [2024-04-22 16:23:12.191201] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC10-0-0] [lt=4][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-04-22 16:23:12.191466] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC10-0-0] [lt=264][errcode=-4002] refresh sever list failed(ret=-4002) [2024-04-22 16:23:12.191499] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC10-0-0] [lt=32][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-04-22 16:23:12.191615] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=72] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10404, mem_used=79030272) [2024-04-22 16:23:12.191610] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=82] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:12.191617] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=43] gc stale ls task succ [2024-04-22 16:23:12.191784] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1918) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=10] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-22 16:23:12.191769] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-22 16:23:12.191809] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1459) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=15] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:12.191882] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1476) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=71] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:12.191948] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2682) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=24] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:12.191945] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [10543][Blacklist][T0][Y0-0000000000000000-0-0] [lt=40] blacklist_loop exec finished(cost_time=219, is_enabled=true, send_cnt=0) [2024-04-22 16:23:12.192276] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:131) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] ====== checkpoint timer task ====== [2024-04-22 16:23:12.192223] WDIAG [RPC] create (ob_poc_rpc_server.cpp:121) [10459][pnio1][T0][YB42AC175883-000616AB095735EE-0-0] [lt=52][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1808, fly_ts=6460183, send_timestamp=1713774185732034, peer="172.23.88.131:36184", sz=224) [2024-04-22 16:23:12.192444] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.192289] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=56] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:12.192632] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2024-04-22 16:23:12.192705] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=10] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:12.192735] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=16] cannot_recycle_log_size statistics(cannot_recycle_log_size=358576, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:12.192788] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=1][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.192835] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10944][T1_L0_G5][T1][YB42AC175883-000616AB095735EE-0-0] [lt=28][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.192853] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10944][T1_L0_G5][T1][YB42AC175883-000616AB095735EE-0-0] [lt=17][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.192881] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=67] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:12.192942] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10944][T1_L0_G5][T1][YB42AC175883-000616AB095735EE-0-0] [lt=32][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.192972] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=54] PNIO [ratelimit] time: 1713774192192885, bytes: 0, bw: 0.000000 MB/s, add_ts: 7248195, add_bytes: 0 [2024-04-22 16:23:12.192721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10481][EventTimer][T0][Y0-0000000000000000-0-0] [lt=140][errcode=0] task scheduled too delay(scheduled_delay=8480947, expected_run_ts_us_=1713774183711614, ObClockGenerator::getRealClock()=1713774192192627, ret=0, ret="OB_SUCCESS", *this={this:0x7f3ce8bff450, caller:ob_event_history_table_operator.cpp:add_event_to_timer_:410, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$_1677, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f3ce8bff6b0, time_wheel:0x7f3cfe433d80, thread_pool:0x7f3cfe4335e0, is_running:1, time_interval:15.00s, expected_run_time:16:23:03.711, task_priority:1, with_handle_protected:0}) [2024-04-22 16:23:12.193124] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=56][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:2368, tid:10882}, {errcode:-5019, dropped:763, tid:10538}, {errcode:-8001, dropped:142, tid:10495}]) [2024-04-22 16:23:12.193231] WDIAG [STORAGE.TRANS] process (ob_gts_rpc.cpp:90) [10944][T1_L0_G5][T1][YB42AC175883-000616AB095735EE-0-0] [lt=7][errcode=-4038] handle request failed(ret=-4038, ret="OB_NOT_MASTER", arg_={tenant_id:1, srr:[mts=1713774185732010], range_size:1, sender:"172.23.88.131:2882"}) [2024-04-22 16:23:12.193371] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:599) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=23] modify replica success(ret=0, ret="OB_SUCCESS", local_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, table_replica={modify_time_us:1713774183740415, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2024-04-22 16:23:12.193398] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1426) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=42] succ to get rec scn(*this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"L_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:12.193502] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10481][EventTimer][T0][Y0-0000000000000000-0-0] [lt=432][errcode=0] task scheduled too delay(scheduled_delay=8481502, expected_run_ts_us_=1713774183711999, ObClockGenerator::getRealClock()=1713774192193501, ret=0, ret="OB_SUCCESS", *this={this:0x7f3ce8bff730, caller:ob_event_history_table_operator.cpp:add_event_to_timer_:410, function_type:ZN9oceanbase5share27ObEventHistoryTableOperator19add_event_to_timer_ERKNS_6common11ObSqlStringEE6$_1677, timer_running_flag:1, total_running_count:6, func_shared_ptr_.ptr:0x7f3ce8bff990, time_wheel:0x7f3cfe433d80, thread_pool:0x7f3cfe4335e0, is_running:1, time_interval:15.00s, expected_run_time:16:23:03.711, task_priority:1, with_handle_protected:0}) [2024-04-22 16:23:12.193494] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=12] current monitor number(seq_=-1) [2024-04-22 16:23:12.193492] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=93] tx ctx memtable get rec scn(this={ObITable:{this:0x7f3cd10be080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f3cd10be080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:12.193580] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:856) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=32] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-22 16:23:12.193617] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:12.193744] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:599) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=104] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1}) [2024-04-22 16:23:12.193802] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:470) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=31] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2024-04-22 16:23:12.193427] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:221) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=47] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=307) [2024-04-22 16:23:12.193836] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=0, rec_log_ts={val:1713774187210958129, v:0}) [2024-04-22 16:23:12.193844] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-22 16:23:12.193850] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] get rec log scn(service_type_=2, rec_log_ts={val:1713773717210958135, v:0}) [2024-04-22 16:23:12.193873] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:306) [10844][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=17] ====== traversal_flush timer task ====== [2024-04-22 16:23:12.193895] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:164) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713773717210958135, v:0}, checkpoint_scn_in_ls_meta={val:1713773717210958135, v:0}, ls_id={id:1}, service_type="DAS_ID") [2024-04-22 16:23:12.193924] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1579) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-22 16:23:12.193893] INFO pn_ratelimit (group.c:589) [10608][IngressService][T0][Y0-0000000000000000-0-0] [lt=21] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:23:12.193938] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:185) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-22 16:23:12.193954] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:192) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-22 16:23:12.193959] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=9] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-22 16:23:12.193996] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=16] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=98) [2024-04-22 16:23:12.193950] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:12.194008] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:146) [10857][T1_CompactionRe][T1][YB42AC175881-000616AB0B2FEBBE-0-0] [lt=57] InfoPoolResizeTask(cost_ts=6) [2024-04-22 16:23:12.194024] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=60][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:12.194036] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:12.194070] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:12.194067] INFO [MDS] flush (mds_table_mgr.cpp:172) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC0E-0-0] [lt=31] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:12.194137] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:338) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC0E-0-0] [lt=66] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-22 16:23:12.194196] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=15] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-22 16:23:12.194200] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:124) [10856][T1_SSTableGC][T1][YB42AC175881-000616AB0B1FEBBD-0-0] [lt=96] SSTableGCTask(cost_ts=167) [2024-04-22 16:23:12.194209] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=8] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-22 16:23:12.194222] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=11] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=98) [2024-04-22 16:23:12.194334] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=48] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:12.194378] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [10492][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=39] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-04-22 16:23:12.194399] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:40, local_retry_times:40, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:12.194532] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=52] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185632184, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185620912, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:12.194621] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=54][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:12.194626] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:12.194629] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:12.194568] INFO [CLOG] run1 (ob_garbage_collector.cpp:1348) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=32] Garbage Collector is running(seq_=51, gc_interval=10000000) [2024-04-22 16:23:12.194666] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:12.194699] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:501) [10811][T1_SSTableDefra][T1][Y0-0000000000000000-0-0] [lt=34] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=0) [2024-04-22 16:23:12.194711] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:517) [10811][T1_SSTableDefra][T1][Y0-0000000000000000-0-0] [lt=11] successfully defragment data blocks(rewrite_cnt=0) [2024-04-22 16:23:12.194585] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:672) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=56] failed to get resp and destroy client, addr: "172.23.88.131:2882", last_write_ts_=1713774185554575, last_read_ts_=1713774192194512 [2024-04-22 16:23:12.194727] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:697) [10811][T1_SSTableDefra][T1][Y0-0000000000000000-0-0] [lt=5] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2024-04-22 16:23:12.194835] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.194901] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:672) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=136] failed to get resp and destroy client, addr: "172.23.88.130:2882", last_write_ts_=1713774185554620, last_read_ts_=1713774192194897 [2024-04-22 16:23:12.194900] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=89][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.194933] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-04-22 16:23:12.194933] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=64] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:12.194938] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.194954] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.194967] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:12.194957] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-04-22 16:23:12.194977] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=9][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:12.194978] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=20][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-04-22 16:23:12.194983] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.194988] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.194983] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:12.194993] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="172.23.88.129:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.194984] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=5][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-04-22 16:23:12.195006] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=20][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:12.195006] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.23.88.129:2882", server="172.23.88.129:2882", tmp_ret=-4038) [2024-04-22 16:23:12.195009] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=23][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:12.195010] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:12.195012] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:12.195018] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-04-22 16:23:12.195021] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:12.195023] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:12.195025] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:12.195027] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:12.195029] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:12.195031] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:12.195039] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774185662216], gts:1713774187494514341, latest_srr:[mts=1713774185288086]}) [2024-04-22 16:23:12.195032] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:12.195035] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:12.195054] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=21][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:12.195055] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=20][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:12.195058] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:12.195041] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2401) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=53] Dumping All Cache Objs(alloc_obj_list.count()=6, alloc_obj_list=[{obj_id:99, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:175160}, {obj_id:100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:174200}, {obj_id:101, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173640}, {obj_id:102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173992}, {obj_id:103, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176531}, {obj_id:104, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175513}]) [2024-04-22 16:23:12.195066] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2690) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=22] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:12.195062] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:12.195060] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:12.195079] INFO [STORAGE.TRANS] get_gts (ob_gts_source.cpp:268) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11] push gts task success(*task={:""}) [2024-04-22 16:23:12.195080] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=19][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:12.195079] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=17][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:12.195099] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:677) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=6] mark black, addr: "172.23.88.129:2882", last_write_ts_=1713774192195094, last_read_ts_=1713774185554650 [2024-04-22 16:23:12.195138] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:12.195150] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10944][T1_L0_G5][T1][YB42AC175882-000616AB091C306E-0-0] [lt=41][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.195159] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10944][T1_L0_G5][T1][YB42AC175882-000616AB091C306E-0-0] [lt=9][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.195139] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:12.195164] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10944][T1_L0_G5][T1][YB42AC175882-000616AB091C306E-0-0] [lt=5][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.195168] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=28][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:12.195143] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:12.195173] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:12.195173] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=29][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:12.195177] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:12.195177] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:12.195181] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:12.195181] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:12.195186] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=4][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:12.195186] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=5][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:12.195196] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:12.195197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:12.195287] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:12.195331] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=42][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:12.195288] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:12.195335] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:12.195338] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:12.195339] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=49][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:12.195343] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:12.195345] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:12.195353] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:12.195363] INFO [LIB] stat (utility.h:1165) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=364, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:12.195357] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC0F-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2024-04-22 16:23:12.195393] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:23:12.195358] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:12.195398] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:12.195400] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=41][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:12.195402] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:12.195405] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:12.195409] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=4][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:12.195413] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:12.195407] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1460050, start=1713774192193544, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:23:12.195419] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:12.195420] INFO [LIB] stat (utility.h:1165) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT APPEND COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=437, this=0x7f3cd2d6d1d0, extra_info="id:1") [2024-04-22 16:23:12.195417] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f3d1177a050, start=1713774192193549, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:12.195423] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:479) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:12.195426] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=9][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:12.195431] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=4][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:12.195429] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1483) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=79] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-04-22 16:23:12.195521] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:141) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2024-04-22 16:23:12.195528] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:252) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] failed to fetch max rule version(ret=-5019) [2024-04-22 16:23:12.195532] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:95) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2024-04-22 16:23:12.195530] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=52][errcode=-4018] try free send task failed(ret=-4018) [2024-04-22 16:23:12.195576] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1441) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=68] gc_check_member_list_ cost time(ret=0, time_us=928) [2024-04-22 16:23:12.195589] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1713) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=7] execute_gc cost time(ret=0, time_us=0) [2024-04-22 16:23:12.195600] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1636) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-22 16:23:12.195604] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=17][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:12.195610] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1658) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=6] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-04-22 16:23:12.195622] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED7F-0-0] [lt=18][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:12.195679] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1713) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] execute_gc cost time(ret=0, time_us=71) [2024-04-22 16:23:12.195685] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=4] ObSafeDestroyHandler start process [2024-04-22 16:23:12.195668] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10709][T1_CoordTR][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] task scheduled too delay(scheduled_delay=6186821, expected_run_ts_us_=1713774186008845, ObClockGenerator::getRealClock()=1713774192195667, ret=0, ret="OB_SUCCESS", *this={this:0x7f3d13c04e70, caller:ob_failure_detector.cpp:mtl_start:91, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_6, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f3d13c050d0, time_wheel:0x7f3cfbb15dc0, thread_pool:0x7f3cfa5c70f0, is_running:1, time_interval:1.00s, expected_run_time:16:23:06.884, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196002] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.196021] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2024-04-22 16:23:12.196044] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=21][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2024-04-22 16:23:12.196022] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10709][T1_CoordTR][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] task scheduled too delay(scheduled_delay=5298299, expected_run_ts_us_=1713774186897720, ObClockGenerator::getRealClock()=1713774192196021, ret=0, ret="OB_SUCCESS", *this={this:0x7f3cf8b0cb10, caller:ob_leader_coordinator.cpp:mtl_start:104, function_type:ZN9oceanbase10logservice11coordinator19ObLeaderCoordinator9mtl_startERPS2_E4$_64, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f3cf8b0cd70, time_wheel:0x7f3cfbb15dc0, thread_pool:0x7f3cfa5c70f0, is_running:1, time_interval:5.00s, expected_run_time:16:23:06.897, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196044] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.196079] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=34][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2024-04-22 16:23:12.196087] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=7][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2024-04-22 16:23:12.196090] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=40] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:12.196095] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:12.196101] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:12.196101] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, ret=-5019) [2024-04-22 16:23:12.196109] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2024-04-22 16:23:12.196109] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=8][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_service_epoch, db_name=oceanbase) [2024-04-22 16:23:12.196114] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:12.196119] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:12.196125] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:12.196130] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:12.196024] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=32][errcode=-8001] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:50, local_retry_times:50, err_:-8001, err_:"OB_SERVER_IS_INIT", retry_type:1, client_ret:-8001}, need_retry=true) [2024-04-22 16:23:12.196117] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_service_epoch) [2024-04-22 16:23:12.196135] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:12.196134] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=205] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:12.196142] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=22][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:12.196141] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:12.196150] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:12.196152] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:12.196161] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:12.196160] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_service_epoch' doesn't exist [2024-04-22 16:23:12.196167] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:12.196168] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=7][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:12.196173] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:12.196177] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:12.196179] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=6][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:12.196185] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", ret=-5019) [2024-04-22 16:23:12.196186] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=8][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:12.196192] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:12.196193] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:12.196200] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:12.196199] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=114][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:12.196198] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:12.196208] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=8][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:12.196218] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:12.196224] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:12.196206] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:12.196207] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=7][errcode=-8001] result set close failed(ret=-8001) [2024-04-22 16:23:12.196231] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=24][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:12.196229] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:12.196233] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=25][errcode=-8001] failed to close result(close_ret=-8001, ret=-8001) [2024-04-22 16:23:12.196229] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=411][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.196242] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:12.196245] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:12.196250] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:12.196252] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD6-0-0] [lt=7][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, aret=-5019, ret=-5019) [2024-04-22 16:23:12.196257] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:12.196259] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:12.196254] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:12.196263] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:12.196266] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:12.196269] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' , ret=-5019) [2024-04-22 16:23:12.196271] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:12.196269] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:12.196276] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:12.196278] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd148e050, start=1713774192195836, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:12.196279] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=9][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:12.196285] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:12.196283] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:12.196288] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:12.196292] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:943) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:12.196293] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=8][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:12.196298] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=7][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:12.196306] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:12.196315] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:12.196322] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=6][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:12.196328] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=6][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:12.196304] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:12.196335] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:12.196339] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=34][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:12.196345] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:12.196360] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=5][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:12.196315] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=8][errcode=-8001] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-8001, ret=-8001) [2024-04-22 16:23:12.196367] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=7][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' "}, aret=-5019, ret=-5019) [2024-04-22 16:23:12.196365] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:672) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2024-04-22 16:23:12.196374] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:12.196342] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:12.196381] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:12.196384] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=41][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:12.196387] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=6][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:12.196382] WDIAG [STORAGE.DUP_TABLE] refresh_dup_tablet_schema_ (ob_dup_table_util.cpp:97) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] get duplicate ls status info failed(ret=-5019, dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2024-04-22 16:23:12.196392] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:12.196395] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=7][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1484050, start=1713774192195980, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:12.196396] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:148) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] refresh dup table schema failed(ret=-5019) [2024-04-22 16:23:12.196400] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:12.196403] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=7][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:12.196404] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:236) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] scan dup ls to find dup_tablet failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:12.196411] WDIAG [STORAGE.DUP_TABLE] runTimerTask (ob_dup_table_util.cpp:73) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute dup ls scan failed(tmp_ret=-5019) [2024-04-22 16:23:12.196414] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7B-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:12.196410] WDIAG [SHARE] inner_get_service_epoch_ (ob_service_epoch_proxy.cpp:223) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED35-0-0] [lt=5][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_service_epoch WHERE tenant_id = '1' AND name = 'heartbeat_service_epoch' ) [2024-04-22 16:23:12.196478] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10828][T1_OBJLockGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] task scheduled too delay(scheduled_delay=5034614, expected_run_ts_us_=1713774187161856, ObClockGenerator::getRealClock()=1713774192196471, ret=0, ret="OB_SUCCESS", *this={this:0x7f3d13c052d0, caller:ob_table_lock_service.cpp:start:141, function_type:ZN9oceanbase11transaction9tablelock18ObTableLockService25ObOBJLockGarbageCollector5startEvE5$_620, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f3d13c05530, time_wheel:0x7f3cee981890, thread_pool:0x7f3cee9810f0, is_running:1, time_interval:10.00s, expected_run_time:16:23:07.161, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196570] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10705][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=151][errcode=0] task scheduled too delay(scheduled_delay=5190194, expected_run_ts_us_=1713774187006375, ObClockGenerator::getRealClock()=1713774192196569, ret=0, ret="OB_SUCCESS", *this={this:0x7f3cf8b0c830, caller:mds_tenant_service.cpp:mtl_start:133, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_108, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f3cf8b0ca90, time_wheel:0x7f3cfbb17a50, thread_pool:0x7f3cfbb172b0, is_running:1, time_interval:15.00s, expected_run_time:16:23:07.637, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196631] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10713][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] task scheduled too delay(scheduled_delay=6469838, expected_run_ts_us_=1713774185726791, ObClockGenerator::getRealClock()=1713774192196630, ret=0, ret="OB_SUCCESS", *this={this:0x7f3cf8b723b0, caller:election_proposer.cpp:register_renew_lease_task_:269, function_type:ZN9oceanbase4palf8election16ElectionProposer26register_renew_lease_task_EvE4$_86, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f3cf8b72610, time_wheel:0x7f3cfa5cb890, thread_pool:0x7f3cfa5cb0f0, is_running:1, time_interval:500.00ms, expected_run_time:16:23:05.726, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10705][T1_MdsT][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] task scheduled too delay(scheduled_delay=5190364, expected_run_ts_us_=1713774187006352, ObClockGenerator::getRealClock()=1713774192196717, ret=0, ret="OB_SUCCESS", *this={this:0x7f3cf8b0c550, caller:mds_tenant_service.cpp:mtl_start:123, function_type:ZN9oceanbase7storage3mds18ObTenantMdsService9mtl_startERPS2_E5$_106, timer_running_flag:1, total_running_count:2, func_shared_ptr_.ptr:0x7f3cf8b0c7b0, time_wheel:0x7f3cfbb17a50, thread_pool:0x7f3cfbb172b0, is_running:1, time_interval:3.00s, expected_run_time:16:23:07.635, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196717] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10713][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] task scheduled too delay(scheduled_delay=6444088, expected_run_ts_us_=1713774185752629, ObClockGenerator::getRealClock()=1713774192196717, ret=0, ret="OB_SUCCESS", *this={this:0x7f3cf8b72690, caller:election_acceptor.cpp:start:124, function_type:ZN9oceanbase4palf8election16ElectionAcceptor5startEvE3$_1, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f3cf8b728f0, time_wheel:0x7f3cfa5cb890, thread_pool:0x7f3cfa5cb0f0, is_running:1, time_interval:250.00ms, expected_run_time:16:23:05.752, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196721] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10711][T1_CoordTF][T1][Y0-0000000000000000-0-0] [lt=45][errcode=0] task scheduled too delay(scheduled_delay=6587878, expected_run_ts_us_=1713774185608842, ObClockGenerator::getRealClock()=1713774192196721, ret=0, ret="OB_SUCCESS", *this={this:0x7f3cf8b0dce0, caller:ob_failure_detector.cpp:mtl_start:86, function_type:ZN9oceanbase10logservice11coordinator17ObFailureDetector9mtl_startERPS2_E3$_4, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f3cf8b0df40, time_wheel:0x7f3cfba89d50, thread_pool:0x7f3cfa5c78b0, is_running:1, time_interval:100.00ms, expected_run_time:16:23:05.608, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196787] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED38-0-0] [lt=50] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:12.196789] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10713][T1_ElectTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] task scheduled too delay(scheduled_delay=3961158, expected_run_ts_us_=1713774188235630, ObClockGenerator::getRealClock()=1713774192196789, ret=0, ret="OB_SUCCESS", *this={this:0x7f3d13c05c50, caller:election_proposer.cpp:reschedule_or_register_prepare_task_after_:310, function_type:ZN9oceanbase4palf8election16ElectionProposer42reschedule_or_register_prepare_task_after_ElE4$_90, timer_running_flag:1, total_running_count:3, func_shared_ptr_.ptr:0x7f3d13c05eb0, time_wheel:0x7f3cfa5cb890, thread_pool:0x7f3cfa5cb0f0, is_running:1, time_interval:10.00s, expected_run_time:16:23:08.235, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.196798] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [10826][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-04-22 16:23:12.196812] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC30-0-0] [lt=27][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.196824] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED3A-0-0] [lt=7] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:12.196827] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC30-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:12.196840] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10496][RSAsyncTask3][T1][YB42AC175881-000616AB0A1FEC30-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:12.196817] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:480) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=56] socket need_disconn(n=0, errno=11) [2024-04-22 16:23:12.196813] WDIAG [RPC] create (ob_poc_rpc_server.cpp:121) [10460][pnio1][T0][YB42AC175883-000616AB09C7334A-0-0] [lt=44][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=258, fly_ts=6298394, send_timestamp=1713774185898410, peer="172.23.88.131:36174", sz=397) [2024-04-22 16:23:12.196887] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:508) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=61] server connection closed, fd: 187, addr: "172.23.88.131:36172" [2024-04-22 16:23:12.196949] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC42-0-0] [lt=19][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.196968] WDIAG [SHARE] get_tenant_primary_zone_score (ob_primary_zone_util.cpp:654) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC76-0-0] [lt=13][errcode=-4016] zone list count unexpected(ret=-4016) [2024-04-22 16:23:12.196965] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC42-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:12.196992] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:480) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=4] socket need_disconn(n=0, errno=11) [2024-04-22 16:23:12.196991] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC12-0-0] [lt=15][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.196976] WDIAG [SHARE] get_tenant_primary_zone_array (ob_primary_zone_util.cpp:613) [10727][T1_COMMONLSSe][T1][YB42AC175881-000616AB099FEC76-0-0] [lt=7][errcode=-4016] failed to get tenant primary zone score(ret=-4016, ret="OB_ERR_UNEXPECTED", tenant_schema={tenant_id:1, schema_version:1, tenant_name:"sys", zone_list:[cnt:0], primary_zone:"", charset_type:2, locked:false, comment:"", name_case_mode:-1, read_only:false, locality_str:"", zone_replica_attr_array:[cnt:0], primary_zone_array:[], previous_locality_str:"", default_tablegroup_id:18446744073709551615, default_tablegroup_name:"", compatibility_mode:-1, drop_tenant_time:-1, status:0, in_recyclebin:false, arbitration_service_status:{status:3}}) [2024-04-22 16:23:12.197006] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:508) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=12] server connection closed, fd: 188, addr: "172.23.88.130:55308" [2024-04-22 16:23:12.197023] WDIAG pkts_sk_consume (handle_io.t.h:57) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=72][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.197026] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=163] task handle constructed(*this={this:0x7f3ce8455e50, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:12.197020] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC12-0-0] [lt=21] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:12.197023] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [10746][T1_FrzTrigger][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] task scheduled too delay(scheduled_delay=5014114, expected_run_ts_us_=1713774187182907, ObClockGenerator::getRealClock()=1713774192197022, ret=0, ret="OB_SUCCESS", *this={this:0x7f3d13c055b0, caller:ob_tenant_freezer.cpp:start:126, function_type:ZN9oceanbase7storage15ObTenantFreezer5startEvE5$_457, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f3d13c05810, time_wheel:0x7f3cfa5c3890, thread_pool:0x7f3cf9dfd660, is_running:1, time_interval:2.00s, expected_run_time:16:23:07.182, task_priority:1, with_handle_protected:1}) [2024-04-22 16:23:12.197040] INFO eloop_handle_sock_event (eloop.c:107) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO sock destroy: sock=0x7f3cd33f89e8, connection=fd:202:local:172.23.88.129:2882:remote:172.23.88.131:36174, err=61 [2024-04-22 16:23:12.197040] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC12-0-0] [lt=17][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:12.197135] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10944][T1_L0_G5][T1][YB42AC175883-000616AB095735F3-0-0] [lt=4][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197143] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10944][T1_L0_G5][T1][YB42AC175883-000616AB095735F3-0-0] [lt=8][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.197157] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10944][T1_L0_G5][T1][YB42AC175883-000616AB095735F3-0-0] [lt=13][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.197155] WDIAG eloop_delay_warn (debug.h:93) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] PNIO [delay_warn] eloop handle events delay high: 6526581, malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2024-04-22 16:23:12.197194] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO sk_destroy: s=0x7f3cd33f89e8 io=0x7f3d00c04408 [2024-04-22 16:23:12.197239] WDIAG pkts_sk_consume (handle_io.t.h:57) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=131][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.197252] INFO eloop_handle_sock_event (eloop.c:107) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sock destroy: sock=0x7f3d01bf9328, connection=fd:186:local:172.23.88.129:2882:remote:172.23.88.130:55292, err=61 [2024-04-22 16:23:12.197276] WDIAG [RPC] create (ob_poc_rpc_server.cpp:121) [10461][pnio1][T0][YB42AC175882-000616AB096C305B-0-0] [lt=42][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1315, fly_ts=4851890, send_timestamp=1713774187345386, peer="172.23.88.130:55302", sz=207) [2024-04-22 16:23:12.197191] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC61-0-0] [lt=2][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.197313] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC61-0-0] [lt=115] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.197322] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC61-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:12.197347] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_destroy: s=0x7f3d01bf9328 io=0x7f3d01004408 [2024-04-22 16:23:12.197276] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=53][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197365] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=88][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.197367] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10947][T1_L0_G0][T1][YB42AC175883-000616AB09A735CD-0-0] [lt=55][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197371] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:191) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.197373] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10712][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=34] task handle constructed(*this={this:0x7f3ce8455c90, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:12.197391] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.23.88.129:2882", msg={tenant_id:1, srr:[mts=1713774192197178], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:12.197396] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10947][T1_L0_G0][T1][YB42AC175882-000616AB096C305B-0-0] [lt=6][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197401] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.23.88.129:2882", msg={tenant_id:1, srr:[mts=1713774192197178], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:12.197396] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10960][T1_L0_G0][T1][YB42AC175882-000616AB096C305A-0-0] [lt=3][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197407] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] gts statistics(tenant_id=1, gts_rpc_cnt=3, get_gts_cache_cnt=224, get_gts_with_stc_cnt=241, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-22 16:23:12.197415] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713774185662216], gts:1713774187494514341, latest_srr:[mts=1713774192197178]}) [2024-04-22 16:23:12.197422] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:175) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-04-22 16:23:12.197425] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [11020][T1_L0_G10][T1][YB42AC175882-000616AB097C30A8-0-0] [lt=17][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197479] WDIAG pkts_sk_consume (handle_io.t.h:57) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.197491] INFO eloop_handle_sock_event (eloop.c:107) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sock destroy: sock=0x7f3d01bfa048, connection=fd:197:local:172.23.88.129:2882:remote:172.23.88.130:55302, err=61 [2024-04-22 16:23:12.197518] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3d01bfa048 io=0x7f3d00804408 [2024-04-22 16:23:12.197521] WDIAG pkts_sk_consume (handle_io.t.h:57) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.197525] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10945][T1_L0_G5][T1][YB42AC175882-000616AB091C3072-0-0] [lt=20][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197531] INFO eloop_handle_sock_event (eloop.c:107) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3d01a04708, connection=fd:215:local:172.23.88.129:2882:remote:172.23.88.131:36184, err=61 [2024-04-22 16:23:12.197532] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10945][T1_L0_G5][T1][YB42AC175882-000616AB091C3072-0-0] [lt=7][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.197538] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10945][T1_L0_G5][T1][YB42AC175882-000616AB091C3072-0-0] [lt=5][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.197561] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3d01a04708 io=0x7f3d01004408 [2024-04-22 16:23:12.197579] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713774187421454783, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=4776122, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1713774192197576, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-22 16:23:12.197623] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:12.197663] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774192197620) [2024-04-22 16:23:12.197672] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774185669379, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:12.197682] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197688] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:12.197709] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197720] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10945][T1_L0_G5][T1][YB42AC175883-000616AB095735F1-0-0] [lt=3][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197727] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10945][T1_L0_G5][T1][YB42AC175883-000616AB095735F1-0-0] [lt=6][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.197724] WDIAG pkts_sk_consume (handle_io.t.h:57) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.197731] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10945][T1_L0_G5][T1][YB42AC175883-000616AB095735F1-0-0] [lt=3][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.197755] INFO eloop_handle_sock_event (eloop.c:107) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO sock destroy: sock=0x7f3d01ba49e8, connection=fd:206:local:172.23.88.129:2882:remote:172.23.88.131:36178, err=61 [2024-04-22 16:23:12.197734] INFO [PALF] get_log (palf_handle_impl.cpp:3994) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F5A-0-0] [lt=23] submit fetch log task success(ret=0, palf_id=1, server="172.23.88.130:2882", fetch_type=0, prev_lsn={lsn:356459}, start_lsn={lsn:356581}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=7) [2024-04-22 16:23:12.197787] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_destroy: s=0x7f3d01ba49e8 io=0x7f3d00804408 [2024-04-22 16:23:12.197790] INFO [PALF] get_log (palf_handle_impl.cpp:3994) [11430][T1_L0_G1][T1][YB42AC175883-000616AB092732B0-0-0] [lt=27] submit fetch log task success(ret=0, palf_id=1, server="172.23.88.131:2882", fetch_type=0, prev_lsn={lsn:358454}, start_lsn={lsn:358576}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=7) [2024-04-22 16:23:12.197824] INFO eloop_handle_sock_event (eloop.c:95) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3cd24d9358, connection=fd:213:local:172.23.88.129:52416:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.197845] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [11015][PxTargetMgr0][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197843] WDIAG [PALF] check_epoch_is_same_with_election (log_state_mgr.cpp:1040) [10906][T1_L0_G9][T1][YB42AC175882-000616AB098C2FFA-0-0] [lt=38][errcode=0] election is not leader now(ret=0, curr_election_role=2, curr_election_epoch=0) [2024-04-22 16:23:12.197860] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=3][errcode=0] server is initiating(server_id=0, local_seq=14849, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:12.197863] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:181) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=14] dummy location not credible, need refresh [2024-04-22 16:23:12.197860] WDIAG [PALF] need_update_leader_ (log_state_mgr.cpp:958) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=128][errcode=0] leader or epoch has changed, need update(bool_ret=true, new_leader="0.0.0.0:0", palf_id=1, new_leader_epoch=0, leader_="172.23.88.129:2882", leader_epoch_=10, self="172.23.88.129:2882") [2024-04-22 16:23:12.197871] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:232) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=7] waiting for refresh locatition cache for target_monitor(tenant_id_=1, refresh_ctrl=12) [2024-04-22 16:23:12.197879] INFO eloop_handle_sock_event (eloop.c:95) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock destroy: sock=0x7f3d01a769e8, connection=fd:225:local:172.23.88.129:59664:remote:0.0.0.0:0, s->mask=0x4000001d [2024-04-22 16:23:12.197883] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO sk_destroy: s=0x7f3cd24d9358 io=0x7f3d01184608 [2024-04-22 16:23:12.197883] INFO [PALF] freeze_pending_log_ (log_sliding_window.cpp:2396) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] freeze_pending_log success(ret=0, palf_id=1, self="172.23.88.129:2882", last_lsn={lsn:358698}) [2024-04-22 16:23:12.197884] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] refresh dummy location cache(ret=0) [2024-04-22 16:23:12.197891] INFO [PALF] to_follower (log_group_buffer.cpp:371) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] to_follower finished(ret=0, available_buffer_size=41943040, reserved_buffer_size=41943040) [2024-04-22 16:23:12.197901] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3d01a769e8 io=0x7f3d00984608 [2024-04-22 16:23:12.197905] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO resp_cb on sk_destroy: packet_id=23074 s=0x7f3d01a769e8 [2024-04-22 16:23:12.197908] INFO [PALF] reset_match_lsn_map_ (log_sliding_window.cpp:3987) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=4] reset_match_lsn_map_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", member_list=3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, max_flushed_end_lsn={lsn:358576}, committed_end_lsn={lsn:358576}) [2024-04-22 16:23:12.197919] INFO [PALF] to_follower_pending (log_sliding_window.cpp:2424) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] to_follower_pending success(ret=0, palf_id=1, self="172.23.88.129:2882", last_lsn={lsn:358698}, max_flushed_end_lsn={lsn:358576}, curr_end_lsn={lsn:358698}, is_all_freezed=true, this={palf_id:1, self:"172.23.88.129:2882", lsn_allocator:{max_log_id:2942, max_lsn:358698, max_scn:{val:1713774187494514341, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:358576}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:358576}, last_submit_end_lsn:{lsn:358698}, last_submit_log_id:2942, last_submit_log_pid:7, max_flushed_lsn:{lsn:358454}, max_flushed_end_lsn:{lsn:358576}, max_flushed_log_pid:7, committed_end_lsn:{lsn:358576}, last_slide_log_id:2941, last_slide_scn:{val:1713774187421454783, v:0}, last_slide_lsn:{lsn:358454}, last_slide_end_lsn:{lsn:358576}, last_slide_log_pid:7, last_slide_log_accum_checksum:1087531090, last_fetch_end_lsn:{lsn:42299499}, last_fetch_max_log_id:4972, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713774179963810, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f3cd2de4530}) [2024-04-22 16:23:12.197945] INFO [PALF] to_follower_pending_ (log_state_mgr.cpp:762) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] to_follower_pending_(ret=0, palf_id=1, pending_end_lsn={lsn:358698}) [2024-04-22 16:23:12.197946] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323C-0-0] [lt=31][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:12.197954] INFO [PALF] reset_status_ (log_state_mgr.cpp:721) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=4] reset_status_(palf_id=1, self="172.23.88.129:2882", leader_epoch=-1, leader_="0.0.0.0:0") [2024-04-22 16:23:12.197953] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323C-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:12.197972] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2024-04-22 16:23:12.197969] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323C-0-0] [lt=15][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:12.197980] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5] task handle constructed(*this={this:0x7f3ce7dcf290, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:12.197977] WDIAG [SHARE] set_default_timeout_ctx (ob_share_util.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=68][errcode=-4012] timeouted(ret=-4012, ret="OB_TIMEOUT", abs_timeout_ts=1713774185741140, ctx_timeout_ts=1713774185741140, worker_timeout_ts=1713774185741139, default_timeout=1000000) [2024-04-22 16:23:12.197978] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323C-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:12.197994] WDIAG [SQL.EXE] get_common_rpc (ob_task_executor_ctx.cpp:213) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=53][errcode=-4012] execute task timeout(timeout=-5997619, ret=-4012) [2024-04-22 16:23:12.197999] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1236) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=21][errcode=-4012] fail to set default_timeout_ctx(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:12.198005] INFO eloop_handle_sock_event (eloop.c:95) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO sock destroy: sock=0x7f3cd24d89b8, connection=fd:203:local:172.23.88.129:59690:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.198005] WDIAG [SQL.EXE] get_common_rpc (ob_task_executor_ctx.cpp:183) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=10][errcode=-4012] get common rpc problem (ret=-4012) [2024-04-22 16:23:12.198005] WDIAG [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:428) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=5][errcode=-4012] batch renew cache failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, tablet_list=[{id:1}], ls_ids=[{id:1}]) [2024-04-22 16:23:12.198016] INFO [SHARE] add_event_with_retry (ob_event_history_table_operator.h:374) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6] add_event_with_retry success(ret=0, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774192197963), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER ACTIVE', 'CURRENT', 'FOLLOWER PENDING', '', 0, '', 0, 'pending_end_lsn:{lsn:358698}', '172.23.88.129', 2882)) [2024-04-22 16:23:12.198011] INFO eloop_handle_sock_event (eloop.c:95) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock destroy: sock=0x7f3d01ba4048, connection=fd:211:local:172.23.88.129:52414:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.198020] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3cd24d89b8 io=0x7f3d01184608 [2024-04-22 16:23:12.198011] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1239) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=5][errcode=-4006] get common rpc proxy failed(task_exec_ctx=0x7f3cfd2ceef8) [2024-04-22 16:23:12.198019] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4725) [2024-04-22 16:23:12.198025] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO resp_cb on sk_destroy: packet_id=23079 s=0x7f3cd24d89b8 [2024-04-22 16:23:12.198023] INFO [PALF] leader_active_to_follower_pending_ (log_state_mgr.cpp:709) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5] [PALF_EVENT] leader_active_to_follower_pending_(palf_id=1, ret=0, self="172.23.88.129:2882", role=2, state=4, pending_end_lsn={lsn:358698}) [2024-04-22 16:23:12.198031] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO sk_destroy: s=0x7f3d01ba4048 io=0x7f3d00984608 [2024-04-22 16:23:12.198029] WDIAG [SQL.DAS] force_refresh_location_cache (ob_das_location_router.cpp:1194) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=7][errcode=-4012] batch renew tablet locations failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, err_no=-4725, is_nonblock=false, failed_list=[{id:1}]) [2024-04-22 16:23:12.198029] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5] switch_state(ret=0, palf_id=1, old_role=1, role=2, old_state=2, state=4, old_leader="172.23.88.129:2882", leader="0.0.0.0:0", old_leader_epoch=10, leader_epoch=-1, need_next_loop=true, diff=151) [2024-04-22 16:23:12.198035] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO resp_cb on sk_destroy: packet_id=23081 s=0x7f3d01ba4048 [2024-04-22 16:23:12.198037] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:789) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:358576}, pending_end_lsn={lsn:358698}) [2024-04-22 16:23:12.198037] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=6][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:61, local_retry_times:61, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:12.198041] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:12.198042] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=4] switch_state(ret=0, palf_id=1, old_role=2, role=2, old_state=4, state=4, old_leader="0.0.0.0:0", leader="0.0.0.0:0", old_leader_epoch=-1, leader_epoch=-1, need_next_loop=false, diff=6) [2024-04-22 16:23:12.198091] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:270) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323C-0-0] [lt=58][errcode=-4122] address in blacklist(ret=-4122, addr="172.23.88.129:2882") [2024-04-22 16:23:12.198098] WDIAG [RPC] post (ob_poc_rpc_proxy.h:231) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323C-0-0] [lt=6][errcode=-4122] check_blacklist failed(addr="172.23.88.129:2882") [2024-04-22 16:23:12.198102] INFO eloop_handle_sock_event (eloop.c:95) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sock destroy: sock=0x7f3cd33ff358, connection=fd:216:local:172.23.88.129:52418:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.198108] INFO [SHARE] add_event (ob_event_history_table_operator.h:261) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=15] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774192198027), 'sql', 'execute_cmd', 'cmd_type', 177, 'sql_text', X'2A2A2A', 'return_code', -4006, 'tenant_id', 1, '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:12.198116] WDIAG [SQL] open_cmd (ob_result_set.cpp:99) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=7][errcode=-4006] execute cmd failed(ret=-4006) [2024-04-22 16:23:12.198121] WDIAG [SQL] open (ob_result_set.cpp:158) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=4][errcode=-4006] execute plan failed(ret=-4006) [2024-04-22 16:23:12.198125] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=3][errcode=-4006] open result set failed(ret=-4006) [2024-04-22 16:23:12.198103] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323C-0-0] [lt=3][errcode=-4122] LogRpc post_request failed(ret=-4122, palf_id_=1, req={this:0x7f3ccba550c0, BASE:{msg_type:"Prepare Request", id:1, sender:"172.23.88.129:2882", receiver:"172.23.88.129:2882", restart_counter:1, ballot_number:11, debug_ts:{src_construct_ts:"23:12.198059", src_serialize_ts:"00:00.0", dest_deserialize_ts:"00:00.0", dest_process_ts:"00:00.0", process_delay:-1713774192198059}, biggest_min_cluster_version_ever_seen:4.3.0.1}, role:"Follower", is_buffer_valid:true, inner_priority_seed:4096, membership_version:{proposal_id:7, config_seq:8}}, server="172.23.88.129:2882") [2024-04-22 16:23:12.198129] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=3][errcode=-4006] result set open failed(ret=-4006, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}) [2024-04-22 16:23:12.198147] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO sk_destroy: s=0x7f3cd33ff358 io=0x7f3d00d84608 [2024-04-22 16:23:12.198150] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=21][errcode=-4006] execute failed(ret=-4006, tenant_id=1, executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, retry_cnt=51, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:12.198106] INFO [SQL.EXE] end_stmt (ob_sql_trans_control.cpp:1139) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=6] end stmt(ret=0, tx_id=0, plain_select=true, stmt_type=1, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185615134, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185591039, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}, rollback=true, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, exec_ctx.get_errcode()=-4725) [2024-04-22 16:23:12.198162] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=6][errcode=-4006] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:51, local_retry_times:51, err_:-4006, err_:"OB_NOT_INIT", retry_type:0, client_ret:-4006}, need_retry=false) [2024-04-22 16:23:12.198166] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=53][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:12.198170] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=6][errcode=-4006] result set close failed(ret=-4006) [2024-04-22 16:23:12.198171] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=4][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:12.198173] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=3][errcode=-4006] result set close failed(ret=-4006) [2024-04-22 16:23:12.198174] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=3][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:12.198177] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=3][errcode=-4006] failed to close result(close_ret=-4006, ret=-4006) [2024-04-22 16:23:12.198188] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10495][RSAsyncTask2][T1][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=3][errcode=-4006] failed to process record(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, record_ret=-4006, ret=-4006) [2024-04-22 16:23:12.198096] WDIAG eloop_delay_warn (debug.h:93) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] PNIO [delay_warn] eloop handle events delay high: 6527582, malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2024-04-22 16:23:12.198193] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10495][RSAsyncTask2][T0][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=4] slow inner sql(last_ret=-4006, sql={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, process_time=7995936) [2024-04-22 16:23:12.198194] WDIAG pkts_sk_consume (handle_io.t.h:57) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.198198] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10495][RSAsyncTask2][T0][YB42AC175881-000616AB0B0FEBE1-0-0] [lt=5][errcode=-4006] failed to process final(executor={ObIExecutor:, sql:"ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'"}, aret=-4006, ret=-4006) [2024-04-22 16:23:12.198205] INFO eloop_handle_sock_event (eloop.c:107) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3cd3372048, connection=fd:192:local:172.23.88.129:2882:remote:172.23.88.130:55288, err=61 [2024-04-22 16:23:12.198212] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10945][T1_L0_G5][T1][YB42AC175882-000616AB091C306F-0-0] [lt=3][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.198219] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10945][T1_L0_G5][T1][YB42AC175882-000616AB091C306F-0-0] [lt=7][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.198220] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=3][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:12.198225] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4006] execute sql failed(ret=-4006, tenant_id=1, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:12.198228] INFO eloop_handle_sock_event (eloop.c:95) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=99] PNIO sock destroy: sock=0x7f3cd33fe048, connection=fd:201:local:172.23.88.129:52412:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.198232] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_destroy: s=0x7f3cd3372048 io=0x7f3d00c04408 [2024-04-22 16:23:12.198246] INFO eloop_handle_sock_event (eloop.c:95) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3cd332d358, connection=fd:229:local:172.23.88.129:59696:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.198253] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10945][T1_L0_G5][T1][YB42AC175882-000616AB091C3071-0-0] [lt=4][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.198242] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4006] retry_while_no_tenant_resource failed(ret=-4006, tenant_id=1) [2024-04-22 16:23:12.198257] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10945][T1_L0_G5][T1][YB42AC175882-000616AB091C3071-0-0] [lt=4][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.198270] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4006] execute_write failed(ret=-4006, tenant_id=1, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881', is_user_sql=false) [2024-04-22 16:23:12.198278] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4006] execute_write failed(ret=-4006, tenant_id=1, sql="ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881'") [2024-04-22 16:23:12.198281] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_destroy: s=0x7f3cd332d358 io=0x7f3d00d84608 [2024-04-22 16:23:12.198283] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4006] execute sql failed(ret=-4006, conn=0x7f3cbe430050, start=1713774184200376, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:12.198285] INFO pktc_resp_cb_on_sk_destroy (pktc_resp.h:41) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO resp_cb on sk_destroy: packet_id=23080 s=0x7f3cd332d358 [2024-04-22 16:23:12.198291] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2024-04-22 16:23:12.198293] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=5][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.198298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="172.23.88.129:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.198304] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.23.88.129:2882", server="172.23.88.129:2882", tmp_ret=-4038) [2024-04-22 16:23:12.198350] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sk_destroy: s=0x7f3cd33fe048 io=0x7f3cffb84608 [2024-04-22 16:23:12.198362] INFO eloop_handle_sock_event (eloop.c:95) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3cd33f8048, connection=fd:224:local:172.23.88.129:59662:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.198366] INFO pktc_sk_new (pktc_sk_factory.h:78) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: s=0x7f3cd33f89e8 [2024-04-22 16:23:12.198375] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3cd33f8048 io=0x7f3cffb84608 [2024-04-22 16:23:12.198386] INFO pktc_sk_new (pktc_sk_factory.h:78) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_new: s=0x7f3cd3372048 [2024-04-22 16:23:12.198467] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC62-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.198483] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC62-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.198472] INFO eloop_handle_sock_event (eloop.c:95) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=38] PNIO sock destroy: sock=0x7f3d01a76048, connection=fd:220:local:172.23.88.129:59658:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.198548] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=43] PNIO sk_destroy: s=0x7f3d01a76048 io=0x7f3cfff84608 [2024-04-22 16:23:12.198531] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC6E-0-0] [lt=88][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.198562] WDIAG [SHARE] store (ob_inner_config_root_addr.cpp:119) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4006] execute sql failed(ret=-4006, sql=ALTER SYSTEM SET rootservice_list = '172.23.88.129:2882:2881;172.23.88.131:2882:2881') [2024-04-22 16:23:12.198563] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC6E-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.198571] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC6E-0-0] [lt=7][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:12.198570] WDIAG [SHARE] store (ob_rs_mgr.cpp:120) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] store rs list failed(agent=0, tmp_ret=-4006, addr_list=[{server:"172.23.88.129:2882", role:1, sql_port:2881}, {server:"172.23.88.131:2882", role:2, sql_port:2881}], force=true) [2024-04-22 16:23:12.198598] WDIAG [RPC] create (ob_poc_rpc_server.cpp:121) [10463][pnio1][T0][YB42AC175882-000616AB07FC2F07-0-0] [lt=5][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1308, fly_ts=4353842, send_timestamp=1713774187844755, peer="172.23.88.130:55290", sz=1507) [2024-04-22 16:23:12.198666] INFO pktc_sk_new (pktc_sk_factory.h:78) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_new: s=0x7f3d01bfa048 [2024-04-22 16:23:12.198681] WDIAG pkts_sk_consume (handle_io.t.h:57) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.198684] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10495][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4006] task process failed, start retry(max retry time=9223372036854775805, retry interval=1000000, ret=-4006) [2024-04-22 16:23:12.198691] INFO eloop_handle_sock_event (eloop.c:107) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3d01bf89b8, connection=fd:194:local:172.23.88.129:2882:remote:172.23.88.130:55290, err=61 [2024-04-22 16:23:12.198689] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=57][errcode=0] server is initiating(server_id=0, local_seq=14850, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:12.198723] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3d01bf89b8 io=0x7f3cffe04408 [2024-04-22 16:23:12.198792] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="172.23.88.129:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.198811] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=18][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.23.88.129:2882", server="172.23.88.129:2882", tmp_ret=-4038) [2024-04-22 16:23:12.198818] INFO pktc_sk_new (pktc_sk_factory.h:78) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: s=0x7f3cd24d89b8 [2024-04-22 16:23:12.198841] INFO pktc_sk_new (pktc_sk_factory.h:78) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sk_new: s=0x7f3cd24d9358 [2024-04-22 16:23:12.198858] INFO pktc_do_connect (pktc_post.h:19) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_new: sk=0x7f3cd3372048, fd=187 [2024-04-22 16:23:12.198874] INFO pktc_do_connect (pktc_post.h:19) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO sk_new: sk=0x7f3d01bfa048, fd=186 [2024-04-22 16:23:12.198880] INFO pktc_do_connect (pktc_post.h:19) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_new: sk=0x7f3cd24d89b8, fd=188 [2024-04-22 16:23:12.198892] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] write client fd succ, fd:188, gid:0x100000005, need_send_negotiation:1 [2024-04-22 16:23:12.198891] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] write client fd succ, fd:187, gid:0x100000006, need_send_negotiation:1 [2024-04-22 16:23:12.198892] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] write client fd succ, fd:186, gid:0x100000001, need_send_negotiation:1 [2024-04-22 16:23:12.198897] INFO eloop_regist (eloop.c:47) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sock regist: 0x7f3cd3372048 fd=187 [2024-04-22 16:23:12.198896] INFO eloop_regist (eloop.c:47) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock regist: 0x7f3cd24d89b8 fd=188 [2024-04-22 16:23:12.198897] INFO eloop_regist (eloop.c:47) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock regist: 0x7f3d01bfa048 fd=186 [2024-04-22 16:23:12.198905] INFO pktc_do_connect (pktc_post.h:19) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO sk_new: sk=0x7f3cd24d9358, fd=194 [2024-04-22 16:23:12.198908] INFO pktc_sk_new (pktc_sk_factory.h:78) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_new: s=0x7f3cd33fe048 [2024-04-22 16:23:12.198908] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock not ready: 0x7f3d01bfa048, fd=186 [2024-04-22 16:23:12.198914] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] write client fd succ, fd:194, gid:0x100000003, need_send_negotiation:1 [2024-04-22 16:23:12.198918] INFO eloop_regist (eloop.c:47) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock regist: 0x7f3cd24d9358 fd=194 [2024-04-22 16:23:12.198936] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock not ready: 0x7f3cd24d9358, fd=194 [2024-04-22 16:23:12.198937] INFO pktc_do_connect (pktc_post.h:19) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: sk=0x7f3cd33f89e8, fd=197 [2024-04-22 16:23:12.198947] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] write client fd succ, fd:197, gid:0x100000002, need_send_negotiation:1 [2024-04-22 16:23:12.198951] INFO eloop_regist (eloop.c:47) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock regist: 0x7f3cd33f89e8 fd=197 [2024-04-22 16:23:12.198953] INFO pktc_do_connect (pktc_post.h:19) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_new: sk=0x7f3cd33fe048, fd=192 [2024-04-22 16:23:12.198960] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] write client fd succ, fd:192, gid:0x100000006, need_send_negotiation:1 [2024-04-22 16:23:12.198961] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock not ready: 0x7f3cd33f89e8, fd=197 [2024-04-22 16:23:12.198966] INFO eloop_regist (eloop.c:47) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sock regist: 0x7f3cd33fe048 fd=192 [2024-04-22 16:23:12.198979] INFO [PALF] handle (fetch_log_engine.cpp:284) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=46] handle fetch_log_task(fetch_log_task={timestamp_us:1713774192197717, id:1, server:"172.23.88.130:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:7, prev_lsn:{lsn:356459}, start_lsn:{lsn:356581}, log_size:41943040, log_count:2048, accepted_mode_pid:7}) [2024-04-22 16:23:12.198968] INFO eloop_handle_sock_event (eloop.c:95) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO sock destroy: sock=0x7f3cd33f7328, connection=fd:221:local:172.23.88.129:59660:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.199034] WDIAG pkts_sk_consume (handle_io.t.h:57) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.199054] INFO eloop_handle_sock_event (eloop.c:107) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO sock destroy: sock=0x7f3d01bf8048, connection=fd:193:local:172.23.88.129:2882:remote:0.0.0.0:0, err=61 [2024-04-22 16:23:12.199050] INFO [PALF] read (log_cache.cpp:100) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1995, hit_cnt=2, read_cnt=2, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:12.199058] WDIAG [RPC] create (ob_poc_rpc_server.cpp:121) [10464][pnio1][T0][YB42AC175882-000616AB098C2FF7-0-0] [lt=3][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=330, fly_ts=6352974, send_timestamp=1713774185846083, peer="172.23.88.130:55272", sz=219) [2024-04-22 16:23:12.199059] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10944][T1_L0_G5][T1][YB42AC175883-000616AB095735F2-0-0] [lt=4][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.199069] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3d01bf8048 io=0x7f3cffe04408 [2024-04-22 16:23:12.199071] ERROR issue_dba_error (ob_log.cpp:1891) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7E-0-0] [lt=12][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4038, file="ob_root_service.cpp", line_no=10753, info="rs_monitor_check : fail to start root service") [2024-04-22 16:23:12.199076] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3768) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x7f3ccc7fd000, next_round_pread_size:2121728, curr_read_pos:1995, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1995, log_storage_:{IteratorStorage:{start_lsn:{lsn:356581}, end_lsn:{lsn:358576}, read_buf:{buf_len_:8192, buf_:0x7f3ccc7fd000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187421454783, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:66, proposal_id:7, committed_lsn:{lsn:358454}, max_scn:{val:1713774187421454783, v:0}, accumulated_checksum:1087531090, log_id:2941, flag:1}}, init_mode_version:0, accumulate_checksum:1087531090, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774187421454783, v:0}}}, replayable_point={val:18446744073709551615, v:3}, prev_lsn={lsn:358454}) [2024-04-22 16:23:12.199114] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=59] PNIO sk_destroy: s=0x7f3cd33f7328 io=0x7f3d01784608 [2024-04-22 16:23:12.199107] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] sock regist: 0x7f3cfd872040 fd=187 [2024-04-22 16:23:12.199138] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] [ussl] sock regist: 0x7f3cfd8074d0 fd=186 [2024-04-22 16:23:12.199146] WDIAG pkts_sk_consume (handle_io.t.h:57) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.199147] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x7f3cfd8726b0 fd=188 [2024-04-22 16:23:12.199138] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3705) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=25] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, prev_lsn={lsn:356459}, iterator={iterator_impl:{buf_:0x7f3ccc7fd000, next_round_pread_size:2121728, curr_read_pos:1995, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1995, log_storage_:{IteratorStorage:{start_lsn:{lsn:356581}, end_lsn:{lsn:358576}, read_buf:{buf_len_:8192, buf_:0x7f3ccc7fd000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187421454783, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:66, proposal_id:7, committed_lsn:{lsn:358454}, max_scn:{val:1713774187421454783, v:0}, accumulated_checksum:1087531090, log_id:2941, flag:1}}, init_mode_version:0, accumulate_checksum:1087531090, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774187421454783, v:0}}}, fetch_log_count=2048, has_fetched_log_count=16) [2024-04-22 16:23:12.199156] INFO eloop_handle_sock_event (eloop.c:107) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3cd33fe9e8, connection=fd:212:local:172.23.88.129:2882:remote:0.0.0.0:0, err=61 [2024-04-22 16:23:12.199156] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x7f3cfd872790 fd=194 [2024-04-22 16:23:12.199179] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] sock regist: 0x7f3cfd872870 fd=197 [2024-04-22 16:23:12.199178] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock not ready: 0x7f3cd24d89b8, fd=188 [2024-04-22 16:23:12.199182] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=17] PNIO sk_destroy: s=0x7f3cd33fe9e8 io=0x7f3cffa04408 [2024-04-22 16:23:12.199180] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=18] submit_change_mode_meta_req success(ret=0, palf_id=1, server="172.23.88.130:2882", mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=7) [2024-04-22 16:23:12.199188] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] sock regist: 0x7f3cfd872950 fd=192 [2024-04-22 16:23:12.199188] INFO [PALF] fetch_log_from_storage (palf_handle_impl.cpp:3490) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FETCH LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=41943040) [2024-04-22 16:23:12.199193] INFO [LIB] stat (utility.h:1165) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=5] [PALF STAT FETCH LOG TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1261, this=0x7f3ce8ff4600, extra_info="") [2024-04-22 16:23:12.199198] INFO [LIB] stat (utility.h:1165) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=5] [PALF STAT FETCH LOG EXECUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=215, this=0x7f3ce8ff4830, extra_info="") [2024-04-22 16:23:12.199202] INFO pktc_sk_new (pktc_sk_factory.h:78) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: s=0x7f3cd33f8048 [2024-04-22 16:23:12.199202] INFO [PALF] handle (fetch_log_engine.cpp:321) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=3] handle fetch log task(ret=0, palf_id=1, handle_cost_time_us=215, fetch_log_task={timestamp_us:1713774192197717, id:1, server:"172.23.88.130:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:7, prev_lsn:{lsn:356459}, start_lsn:{lsn:356581}, log_size:41943040, log_count:2048, accepted_mode_pid:7}, fetch_stat={total_size:1995, group_log_cnt:16, read_cost:0, get_cost:0, send_cost:35}) [2024-04-22 16:23:12.199213] INFO [PALF] handle (fetch_log_engine.cpp:284) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=9] handle fetch_log_task(fetch_log_task={timestamp_us:1713774192197789, id:1, server:"172.23.88.131:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:7, prev_lsn:{lsn:358454}, start_lsn:{lsn:358576}, log_size:41943040, log_count:2048, accepted_mode_pid:7}) [2024-04-22 16:23:12.199216] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] sock regist: 0x7f3cfd872a30 fd=198 [2024-04-22 16:23:12.199212] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC6F-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.199223] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] accept new connection, fd:198, src_addr:172.23.88.130:55316 [2024-04-22 16:23:12.199225] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC6F-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.199223] WDIAG [PALF] next (log_iterator_impl.h:703) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4008] there is no readable log, set next_min_scn to OB_INVALID_TIMESTAMP(ret=-4008, this={buf_:null, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358576}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogGroupEntryHeader:{magic:0, version:0, group_size:0, proposal_id:9223372036854775807, committed_lsn:{lsn:18446744073709551615}, max_scn:{val:18446744073709551615, v:3}, accumulated_checksum:0, log_id:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, info={reason:2, log_scn:{val:18446744073709551615, v:3}}, replayable_point_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:12.199245] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3768) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:null, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358576}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogGroupEntryHeader:{magic:0, version:0, group_size:0, proposal_id:9223372036854775807, committed_lsn:{lsn:18446744073709551615}, max_scn:{val:18446744073709551615, v:3}, accumulated_checksum:0, log_id:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}, replayable_point={val:18446744073709551615, v:3}, prev_lsn={lsn:358454}) [2024-04-22 16:23:12.199256] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] client send negotiation message succ, fd:192, addr:172.23.88.129:59724, auth_method:NONE, gid:0x100000006 [2024-04-22 16:23:12.199257] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=68][errcode=0] server is initiating(server_id=0, local_seq=14851, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:12.199265] INFO pktc_sk_new (pktc_sk_factory.h:78) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO sk_new: s=0x7f3d01ba4048 [2024-04-22 16:23:12.199268] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] give back fd to origin epoll succ, client_fd:192, client_epfd:145, event:0x8000000d, client_addr:172.23.88.131:2882, has_error:0 [2024-04-22 16:23:12.199278] WDIAG pkts_sk_consume (handle_io.t.h:57) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.199260] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3705) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=14] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, prev_lsn={lsn:358454}, iterator={iterator_impl:{buf_:null, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358576}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogGroupEntryHeader:{magic:0, version:0, group_size:0, proposal_id:9223372036854775807, committed_lsn:{lsn:18446744073709551615}, max_scn:{val:18446744073709551615, v:3}, accumulated_checksum:0, log_id:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}, fetch_log_count=2048, has_fetched_log_count=0) [2024-04-22 16:23:12.199287] INFO eloop_handle_sock_event (eloop.c:107) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock destroy: sock=0x7f3cd33f6048, connection=fd:205:local:172.23.88.129:2882:remote:0.0.0.0:0, err=61 [2024-04-22 16:23:12.199299] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sk_destroy: s=0x7f3cd33f6048 io=0x7f3cffa04408 [2024-04-22 16:23:12.199296] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=23] submit_change_mode_meta_req success(ret=0, palf_id=1, server="172.23.88.131:2882", mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=7) [2024-04-22 16:23:12.199294] INFO [CLOG] handle_sync_request (log_request_handler.cpp:513) [11430][T1_L0_G1][T1][YB42AC175883-000616AB094732D4-0-0] [lt=6] i am not leader(ret=-4038, palf_id=1, req={get_type:1}, role=2, is_pending_state=true) [2024-04-22 16:23:12.199307] INFO pktc_do_connect (pktc_post.h:19) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: sk=0x7f3cd33f8048, fd=200 [2024-04-22 16:23:12.199303] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="172.23.88.129:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.199309] INFO pktc_sk_new (pktc_sk_factory.h:78) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: s=0x7f3d01ba49e8 [2024-04-22 16:23:12.199313] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=9][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.23.88.129:2882", server="172.23.88.129:2882", tmp_ret=-4038) [2024-04-22 16:23:12.199316] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] write client fd succ, fd:200, gid:0x100000000, need_send_negotiation:1 [2024-04-22 16:23:12.199320] INFO eloop_regist (eloop.c:47) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock regist: 0x7f3cd33f8048 fd=200 [2024-04-22 16:23:12.199344] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC63-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.199363] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC63-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.199369] INFO pktc_do_connect (pktc_post.h:19) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: sk=0x7f3d01ba49e8, fd=202 [2024-04-22 16:23:12.199372] INFO pktc_do_connect (pktc_post.h:19) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: sk=0x7f3d01ba4048, fd=201 [2024-04-22 16:23:12.199377] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] write client fd succ, fd:202, gid:0x100000001, need_send_negotiation:1 [2024-04-22 16:23:12.199380] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] write client fd succ, fd:201, gid:0x100000005, need_send_negotiation:1 [2024-04-22 16:23:12.199381] INFO eloop_regist (eloop.c:47) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock regist: 0x7f3d01ba49e8 fd=202 [2024-04-22 16:23:12.199384] INFO eloop_regist (eloop.c:47) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock regist: 0x7f3d01ba4048 fd=201 [2024-04-22 16:23:12.199392] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock not ready: 0x7f3d01ba49e8, fd=202 [2024-04-22 16:23:12.199394] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock connect OK: 0x7f3cd33fe048 fd:192:local:172.23.88.129:59724:remote:172.23.88.131:2882 [2024-04-22 16:23:12.199396] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] client send negotiation message succ, fd:197, addr:172.23.88.129:59722, auth_method:NONE, gid:0x100000002 [2024-04-22 16:23:12.199413] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] give back fd to origin epoll succ, client_fd:197, client_epfd:117, event:0x8000000d, client_addr:172.23.88.131:2882, has_error:0 [2024-04-22 16:23:12.199444] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock not ready: 0x7f3cd3372048, fd=187 [2024-04-22 16:23:12.199448] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock connect OK: 0x7f3cd33f89e8 fd:197:local:172.23.88.129:59722:remote:172.23.88.131:2882 [2024-04-22 16:23:12.199449] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock not ready: 0x7f3d01ba4048, fd=201 [2024-04-22 16:23:12.199387] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4F-0-0] [lt=8][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.199453] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock not ready: 0x7f3cd3372048, fd=187 [2024-04-22 16:23:12.199429] WDIAG [RPC] create (ob_poc_rpc_server.cpp:121) [10458][pnio1][T0][YB42AC175882-000616AB08EC2F59-0-0] [lt=2][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=5379, fly_ts=6263302, send_timestamp=1713774185936124, peer="172.23.88.130:55312", sz=247) [2024-04-22 16:23:12.199428] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] client send negotiation message succ, fd:194, addr:172.23.88.129:52426, auth_method:NONE, gid:0x100000003 [2024-04-22 16:23:12.199469] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4F-0-0] [lt=69] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:12.199479] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC4F-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:12.199491] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] give back fd to origin epoll succ, client_fd:194, client_epfd:124, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:12.199505] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO sock connect OK: 0x7f3cd24d9358 fd:194:local:172.23.88.129:52426:remote:172.23.88.130:2882 [2024-04-22 16:23:12.199514] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] client send negotiation message succ, fd:188, addr:172.23.88.129:52424, auth_method:NONE, gid:0x100000005 [2024-04-22 16:23:12.199522] WARN update_fail_count (ob_root_service.cpp:10758) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7E-0-0] [lt=434][errcode=-4752] rootservice start process has failure(msg="rootservice start()/do_restart() has failure", ret=-4038, ret="OB_NOT_MASTER", fail_cnt=11) [2024-04-22 16:23:12.199548] WDIAG pkts_sk_consume (handle_io.t.h:57) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.199552] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] give back fd to origin epoll succ, client_fd:188, client_epfd:138, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:12.199559] INFO eloop_handle_sock_event (eloop.c:107) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3cd1aac9b8, connection=fd:195:local:172.23.88.129:2882:remote:172.23.88.130:55312, err=61 [2024-04-22 16:23:12.199569] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10494][RSAsyncTask1][T0][YB42AC175881-000616AB0A0FEC7E-0-0] [lt=3][errcode=-4038] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-4038) [2024-04-22 16:23:12.199568] INFO [PALF] handle (fetch_log_engine.cpp:284) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=7] handle fetch_log_task(fetch_log_task={timestamp_us:1713774192199551, id:1, server:"172.23.88.130:2882", fetch_type:"FETCH_LOG_FOLLOWER", proposal_id:7, prev_lsn:{lsn:356459}, start_lsn:{lsn:356581}, log_size:41943040, log_count:2048, accepted_mode_pid:7}) [2024-04-22 16:23:12.199560] INFO [PALF] get_log (palf_handle_impl.cpp:3994) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F59-0-0] [lt=11] submit fetch log task success(ret=0, palf_id=1, server="172.23.88.130:2882", fetch_type=0, prev_lsn={lsn:356459}, start_lsn={lsn:356581}, fetch_log_size=41943040, fetch_log_count=2048, accepted_mode_pid=7) [2024-04-22 16:23:12.199591] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO sk_destroy: s=0x7f3cd1aac9b8 io=0x7f3d01604408 [2024-04-22 16:23:12.199596] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] start to print blacklist info [2024-04-22 16:23:12.199602] INFO eloop_handle_sock_event (eloop.c:95) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock destroy: sock=0x7f3cd33f69b8, connection=fd:207:local:172.23.88.129:2882:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.199601] INFO [CLOG] handle_sync_request (log_request_handler.cpp:513) [11430][T1_L0_G1][T1][YB42AC175882-000616AB090C2F2F-0-0] [lt=17] i am not leader(ret=-4038, palf_id=1, req={get_type:1}, role=2, is_pending_state=true) [2024-04-22 16:23:12.199591] WDIAG [PALF] batch_fetch_log_each_round_ (palf_handle_impl.cpp:3768) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4008] iterator next failed(ret=-4008, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x7f3ccc7f7000, next_round_pread_size:2121728, curr_read_pos:1995, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1995, log_storage_:{IteratorStorage:{start_lsn:{lsn:356581}, end_lsn:{lsn:358576}, read_buf:{buf_len_:8192, buf_:0x7f3ccc7f7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187421454783, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:66, proposal_id:7, committed_lsn:{lsn:358454}, max_scn:{val:1713774187421454783, v:0}, accumulated_checksum:1087531090, log_id:2941, flag:1}}, init_mode_version:0, accumulate_checksum:1087531090, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774187421454783, v:0}}}, replayable_point={val:18446744073709551615, v:3}, prev_lsn={lsn:358454}) [2024-04-22 16:23:12.199606] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock connect OK: 0x7f3cd24d89b8 fd:188:local:172.23.88.129:52424:remote:172.23.88.130:2882 [2024-04-22 16:23:12.199615] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3cd33f69b8 io=0x7f3d01604408 [2024-04-22 16:23:12.199596] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6E-0-0] [lt=70][errcode=0] execute rpc failed(rc=-4012, server="172.23.88.131:2882", timeout=2000000, packet code=1193, arg={addr:"172.23.88.129:2882"}) [2024-04-22 16:23:12.199628] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6E-0-0] [lt=32][errcode=0] execute rpc failed(rc=-4012, server="172.23.88.130:2882", timeout=2000000, packet code=1193, arg={addr:"172.23.88.129:2882"}) [2024-04-22 16:23:12.199622] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] client send negotiation message succ, fd:186, addr:172.23.88.129:52422, auth_method:NONE, gid:0x100000001 [2024-04-22 16:23:12.199636] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:875) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6E-0-0] [lt=5][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.131:2882") [2024-04-22 16:23:12.199641] INFO pktc_sk_new (pktc_sk_factory.h:78) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: s=0x7f3cd1aac9b8 [2024-04-22 16:23:12.199630] INFO [PALF] fetch_log_from_storage_ (palf_handle_impl.cpp:3705) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=22] fetch_log_from_storage_ finished(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, prev_lsn={lsn:356459}, iterator={iterator_impl:{buf_:0x7f3ccc7f7000, next_round_pread_size:2121728, curr_read_pos:1995, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1995, log_storage_:{IteratorStorage:{start_lsn:{lsn:356581}, end_lsn:{lsn:358576}, read_buf:{buf_len_:8192, buf_:0x7f3ccc7f7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187421454783, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:66, proposal_id:7, committed_lsn:{lsn:358454}, max_scn:{val:1713774187421454783, v:0}, accumulated_checksum:1087531090, log_id:2941, flag:1}}, init_mode_version:0, accumulate_checksum:1087531090, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774187421454783, v:0}}}, fetch_log_count=2048, has_fetched_log_count=16) [2024-04-22 16:23:12.199643] WDIAG [SHARE.LOCATION] detect_ls_leaders_ (ob_ls_location_service.cpp:875) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6E-0-0] [lt=7][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.23.88.130:2882") [2024-04-22 16:23:12.199644] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO [ratelimit] time: 1713774192199643, bytes: 66096757, bw: 0.030637 MB/s, add_ts: 7278415, add_bytes: 233819 [2024-04-22 16:23:12.199655] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO sock not ready: 0x7f3cd33f8048, fd=200 [2024-04-22 16:23:12.199659] INFO [PALF] submit_fetch_mode_meta_resp (log_mode_mgr.cpp:826) [10718][T1_FetchLog0][T1][Y0-0000000000000000-0-0] [lt=16] submit_change_mode_meta_req success(ret=0, palf_id=1, server="172.23.88.130:2882", mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_pid=7) [2024-04-22 16:23:12.199664] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock not ready: 0x7f3cd33f8048, fd=200 [2024-04-22 16:23:12.199647] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=6] ls blacklist refresh finish(cost_time=3345) [2024-04-22 16:23:12.199659] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] give back fd to origin epoll succ, client_fd:186, client_epfd:110, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:12.199706] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] client send negotiation message succ, fd:187, addr:172.23.88.129:52420, auth_method:NONE, gid:0x100000006 [2024-04-22 16:23:12.199711] INFO pktc_do_connect (pktc_post.h:19) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: sk=0x7f3cd1aac9b8, fd=193 [2024-04-22 16:23:12.199710] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=3] PNIO sock connect OK: 0x7f3d01bfa048 fd:186:local:172.23.88.129:52422:remote:172.23.88.130:2882 [2024-04-22 16:23:12.199723] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] write client fd succ, fd:193, gid:0x100000002, need_send_negotiation:1 [2024-04-22 16:23:12.199730] INFO eloop_regist (eloop.c:47) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO sock regist: 0x7f3cd1aac9b8 fd=193 [2024-04-22 16:23:12.199733] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] give back fd to origin epoll succ, client_fd:187, client_epfd:145, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:12.199738] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock not ready: 0x7f3cd1aac9b8, fd=193 [2024-04-22 16:23:12.199756] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock connect OK: 0x7f3cd3372048 fd:187:local:172.23.88.129:52420:remote:172.23.88.130:2882 [2024-04-22 16:23:12.199764] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:111) [10528][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=61][errcode=-4015] clock out of order(cur_ts=1713774192199740, cur_ts_=1713774185670275, delta=6529465) [2024-04-22 16:23:12.199794] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x7f3cfd8074d0 fd=200 [2024-04-22 16:23:12.199818] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] sock regist: 0x7f3cfd872040 fd=202 [2024-04-22 16:23:12.199829] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x7f3cfd872b00 fd=201 [2024-04-22 16:23:12.199841] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x7f3cfd872be0 fd=193 [2024-04-22 16:23:12.199894] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] client send negotiation message succ, fd:193, addr:172.23.88.129:52438, auth_method:NONE, gid:0x100000002 [2024-04-22 16:23:12.199916] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] give back fd to origin epoll succ, client_fd:193, client_epfd:117, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:12.199936] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] client send negotiation message succ, fd:201, addr:172.23.88.129:59728, auth_method:NONE, gid:0x100000005 [2024-04-22 16:23:12.199937] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=5] PNIO sock connect OK: 0x7f3cd1aac9b8 fd:193:local:172.23.88.129:52438:remote:172.23.88.130:2882 [2024-04-22 16:23:12.199966] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] give back fd to origin epoll succ, client_fd:201, client_epfd:138, event:0x8000000d, client_addr:172.23.88.131:2882, has_error:0 [2024-04-22 16:23:12.199882] WDIAG eloop_delay_warn (debug.h:93) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] PNIO [delay_warn] eloop handle events delay high: 6529582, malloc=0/0 write=0/0 read=0/0 server_process=0/0 client_cb=0/0 [2024-04-22 16:23:12.199989] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO sock connect OK: 0x7f3d01ba4048 fd:201:local:172.23.88.129:59728:remote:172.23.88.131:2882 [2024-04-22 16:23:12.199994] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] client send negotiation message succ, fd:202, addr:172.23.88.129:59730, auth_method:NONE, gid:0x100000001 [2024-04-22 16:23:12.200010] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] give back fd to origin epoll succ, client_fd:202, client_epfd:110, event:0x8000000d, client_addr:172.23.88.131:2882, has_error:0 [2024-04-22 16:23:12.200036] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO sock connect OK: 0x7f3d01ba49e8 fd:202:local:172.23.88.129:59730:remote:172.23.88.131:2882 [2024-04-22 16:23:12.200040] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA4-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.200028] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] client send negotiation message succ, fd:200, addr:172.23.88.129:52432, auth_method:NONE, gid:0x100000000 [2024-04-22 16:23:12.200064] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10479][EvtHisUpdTask][T1][YB42AC175881-000616AB09EFECA4-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.200081] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=41] [ussl] give back fd to origin epoll succ, client_fd:200, client_epfd:103, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:12.200116] INFO eloop_handle_sock_event (eloop.c:95) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=95] PNIO sock destroy: sock=0x7f3cd33f9388, connection=fd:231:local:172.23.88.129:59670:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.200124] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock connect OK: 0x7f3cd33f8048 fd:200:local:172.23.88.129:52432:remote:172.23.88.130:2882 [2024-04-22 16:23:12.200142] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="172.23.88.129:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.200162] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=19][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="172.23.88.129:2882", server="172.23.88.129:2882", tmp_ret=-4038) [2024-04-22 16:23:12.200274] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO sk_destroy: s=0x7f3cd33f9388 io=0x7f3d00584608 [2024-04-22 16:23:12.200330] INFO eloop_handle_sock_event (eloop.c:95) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO sock destroy: sock=0x7f3cd33d4048, connection=fd:189:local:172.23.88.129:52410:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.200353] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO sk_destroy: s=0x7f3cd33d4048 io=0x7f3d00584608 [2024-04-22 16:23:12.200409] INFO pktc_sk_new (pktc_sk_factory.h:78) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: s=0x7f3cd33d4048 [2024-04-22 16:23:12.200529] WDIAG [PALF] process (log_rpc_proxy.cpp:44) [10464][pnio1][T0][YB42AC175881-000616AB0A2FF4E9-0-0] [lt=13][errcode=-4109] there is an rpc error in logservice(rcode={code:-4109, msg:"", warnings:[]}, dst="172.23.88.130:2882", pcode=5377) [2024-04-22 16:23:12.200645] INFO pktc_do_connect (pktc_post.h:19) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO sk_new: sk=0x7f3cd33d4048, fd=189 [2024-04-22 16:23:12.200672] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] write client fd succ, fd:189, gid:0x100000004, need_send_negotiation:1 [2024-04-22 16:23:12.200678] INFO eloop_regist (eloop.c:47) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO sock regist: 0x7f3cd33d4048 fd=189 [2024-04-22 16:23:12.200693] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=56] [ussl] sock regist: 0x7f3cfd872040 fd=189 [2024-04-22 16:23:12.200734] INFO pktc_sk_new (pktc_sk_factory.h:78) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_new: s=0x7f3cd1aad358 [2024-04-22 16:23:12.200749] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] client send negotiation message succ, fd:189, addr:172.23.88.129:59734, auth_method:NONE, gid:0x100000004 [2024-04-22 16:23:12.200764] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] give back fd to origin epoll succ, client_fd:189, client_epfd:131, event:0x8000000d, client_addr:172.23.88.131:2882, has_error:0 [2024-04-22 16:23:12.200862] INFO pktc_do_connect (pktc_post.h:19) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO sk_new: sk=0x7f3cd1aad358, fd=195 [2024-04-22 16:23:12.200891] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] write client fd succ, fd:195, gid:0x100000004, need_send_negotiation:1 [2024-04-22 16:23:12.200899] INFO eloop_regist (eloop.c:47) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO sock regist: 0x7f3cd1aad358 fd=195 [2024-04-22 16:23:12.200947] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x7f3cfd8074d0 fd=195 [2024-04-22 16:23:12.200976] WDIAG [RPC] create (ob_poc_rpc_server.cpp:121) [10462][pnio1][T0][YB42AC175882-000616AB091C306D-0-0] [lt=4][errcode=0] PNIO packet wait too much time between proxy and server_cb(pcode=1808, fly_ts=6354497, send_timestamp=1713774185846474, peer="172.23.88.130:55310", sz=224) [2024-04-22 16:23:12.201008] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] client send negotiation message succ, fd:195, addr:172.23.88.129:52442, auth_method:NONE, gid:0x100000004 [2024-04-22 16:23:12.201025] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] give back fd to origin epoll succ, client_fd:195, client_epfd:131, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:23:12.201132] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10943][T1_L0_G5][T1][YB42AC175882-000616AB091C306D-0-0] [lt=15][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.201147] WDIAG [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:142) [10944][T1_L0_G5][T1][YB42AC175882-000616AB091C3070-0-0] [lt=8][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.201153] WDIAG pkts_sk_consume (handle_io.t.h:57) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.201180] INFO eloop_handle_sock_event (eloop.c:107) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO sock destroy: sock=0x7f3cd3205358, connection=fd:208:local:172.23.88.129:2882:remote:172.23.88.130:55310, err=61 [2024-04-22 16:23:12.201258] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3cd3205358 io=0x7f3d00404408 [2024-04-22 16:23:12.201276] INFO eloop_handle_sock_event (eloop.c:95) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO sock destroy: sock=0x7f3cd1aac048, connection=fd:214:local:172.23.88.129:2882:remote:0.0.0.0:0, s->mask=0x1d [2024-04-22 16:23:12.201297] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sk_destroy: s=0x7f3cd1aac048 io=0x7f3d00404408 [2024-04-22 16:23:12.201330] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO sock connect OK: 0x7f3cd1aad358 fd:195:local:172.23.88.129:52442:remote:172.23.88.130:2882 [2024-04-22 16:23:12.201350] WDIAG [COORDINATOR] compare_with (election_priority_impl.cpp:212) [10917][T1_L0_G2][T1][YB42AC175883-000616AB0A37323D-0-0] [lt=31][errcode=0] compare between invalid priority [2024-04-22 16:23:12.201396] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=4] PNIO sock connect OK: 0x7f3cd33d4048 fd:189:local:172.23.88.129:59734:remote:172.23.88.131:2882 [2024-04-22 16:23:12.201812] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4638] execute rpc fail(addr="172.23.88.129:2882", pcode=258, ret=-4638, timeout=2000000) [2024-04-22 16:23:12.201835] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4638] [2024-04-22 16:23:12.201856] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:12.201874] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:12.201880] WDIAG [SERVER] do_renew_lease (ob_lease_state_mgr.cpp:414) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4122] can't get lease from rs(rs_addr="172.23.88.129:2882", ret=-4122) [2024-04-22 16:23:12.201887] WDIAG [SERVER] renew_lease (ob_lease_state_mgr.cpp:261) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4122] do_renew_lease failed(ret=-4122) [2024-04-22 16:23:12.201892] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4] [RS_MGR] new master rootserver found(rootservice="172.23.88.129:2882", cluster_id=1713772869) [2024-04-22 16:23:12.201896] INFO [SERVER] renew_lease (ob_lease_state_mgr.cpp:271) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4] renew_master_rootserver successfully, try renew lease again [2024-04-22 16:23:12.201917] INFO [SHARE.PT] update_follower_replica_ (ob_inmemory_ls_table.cpp:170) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=4] add follower replica for sys ls(new_replica={modify_time_us:1713774192201916, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2024-04-22 16:23:12.201939] INFO [SHARE.PT] update_follower_replica_ (ob_inmemory_ls_table.cpp:173) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=22] inmemory ls table add replica(ret=0, ret="OB_SUCCESS", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2024-04-22 16:23:12.201981] INFO [SHARE.PT] update (ob_inmemory_ls_table.cpp:145) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=5] update ls replica(ret=0, ret="OB_SUCCESS", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1713774192201916, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774183894823, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-04-22 16:23:12.202013] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=31] update ls replica(ret=0, ret="OB_SUCCESS", time_used=103, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:7, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2024-04-22 16:23:12.202033] INFO [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:231) [10901][LeaseHB][T1][Y0-0000000000000000-0-0] [lt=12] try report sys log stream succeed [2024-04-22 16:23:12.202115] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4122] check_blacklist failed(ret=-4122) [2024-04-22 16:23:12.202130] WDIAG [SERVER] do_renew_lease (ob_lease_state_mgr.cpp:414) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4122] can't get lease from rs(rs_addr="172.23.88.129:2882", ret=-4122) [2024-04-22 16:23:12.202136] WDIAG [SERVER] renew_lease (ob_lease_state_mgr.cpp:275) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4122] try do_renew_lease again failed, will do it no next heartbeat(ret=-4122) [2024-04-22 16:23:12.202142] TRACE [TRACE] renew_lease (ob_lease_state_mgr.cpp:287) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] [slow heartbeat](TRACE=begin_ts=1713774192191313 2024-04-22 08:23:12.191313|[renew_lease_begin] u=0 |[do_renew_lease_begin] u=56 |[send_heartbeat_begin] u=1710 |[send_heartbeat_end] u=8801 ret:-4122|[do_renew_lease_end] u=6 ret:-4122|[renew_master_rs_begin] u=5 |[renew_lease_end] u=5 |[do_renew_lease_begin] u=146 |[send_heartbeat_begin] u=67 |[send_heartbeat_end] u=20 ret:-4122|[do_renew_lease_end] u=7 ret:-4122|[renew_lease_end] u=5 ret:-4122|total_timeu=10828) [2024-04-22 16:23:12.202387] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=127] Cache replace map node details(ret=0, replace_node_count=0, replace_time=6532392, replace_start_pos=817882, replace_num=62914) [2024-04-22 16:23:12.202430] WDIAG [LIB] ~ObTimeGuard (utility.h:855) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41][errcode=-4389] destruct(*this=time guard 'replace_fragement_node' cost too much time, used=6532475) [2024-04-22 16:23:12.202466] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:12.204780] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10483][LSSysTblUp0][T0][YB42AC175881-000616AB0D0FFC0D-0-0] [lt=24][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774192202026}) [2024-04-22 16:23:12.205335] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x7f3cfd872b00 fd=203 [2024-04-22 16:23:12.205354] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:203, src_addr:172.23.88.131:36226 [2024-04-22 16:23:12.209359] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBE-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.209394] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBE-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_rootservice_event_history, table_name.ptr()="data_size:31, data:5F5F616C6C5F726F6F74736572766963655F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.209404] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBE-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:12.210094] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBF-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.210120] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBBF-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=__all_rootservice_event_history, table_name.ptr()="data_size:31, data:5F5F616C6C5F726F6F74736572766963655F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.214484] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=362672, this={dio_aligned_buf:{buf_write_offset:2346, buf_padding_size:1750, align_size:4096, aligned_buf_size:2125824, aligned_used_ts:5842, truncate_used_ts:2146}, log_block_size:67108864, dir_fd:235, io_fd:237}, aligned_buf_len=4096, aligned_buf="", aligned_block_offset=360448, buf_len=122, total_write_size_=3825, total_write_size_after_dio_=131072, ob_pwrite_used_ts=150107, count=31, each_pwrite_cost=4842) [2024-04-22 16:23:12.214526] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=16779, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:12.214535] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2161) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:358576}], scn_array=[cnt:1, {val:1713774187494514341, v:0}], curr_size=122, accum_size=122, time_cost=16779) [2024-04-22 16:23:12.214551] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=122) [2024-04-22 16:23:12.214566] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=6] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=2318, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:12.214602] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=7, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:12.218974] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10493][RSAsyncTask0][T0][Y0-000616AB09FFEBEB-0-0] [lt=48][errcode=-4038] task process failed, start retry(max retry time=9223372036854775804, retry interval=1000000, ret=-4038) [2024-04-22 16:23:12.218993] INFO [SHARE] run2 (ob_async_task_queue.cpp:169) [10493][RSAsyncTask0][T0][Y0-000616AB09FFEBEB-0-0] [lt=10] async task queue stop [2024-04-22 16:23:12.219037] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10496][RSAsyncTask3][T0][Y0-000616AB0A1FEC31-0-0] [lt=17][errcode=-4038] task process failed, start retry(max retry time=9223372036854775803, retry interval=1000000, ret=-4038) [2024-04-22 16:23:12.219052] INFO [SHARE] run2 (ob_async_task_queue.cpp:169) [10496][RSAsyncTask3][T0][Y0-000616AB0A1FEC31-0-0] [lt=7] async task queue stop [2024-04-22 16:23:12.219096] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10495][RSAsyncTask2][T0][Y0-000616AB0B0FEBE2-0-0] [lt=22][errcode=-4038] task process failed, start retry(max retry time=9223372036854775802, retry interval=1000000, ret=-4038) [2024-04-22 16:23:12.219116] INFO [SHARE] run2 (ob_async_task_queue.cpp:169) [10495][RSAsyncTask2][T0][Y0-000616AB0B0FEBE2-0-0] [lt=11] async task queue stop [2024-04-22 16:23:12.219145] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [10494][RSAsyncTask1][T0][Y0-000616AB0A0FEC7F-0-0] [lt=16][errcode=-4038] task process failed, start retry(max retry time=9223372036854775801, retry interval=1000000, ret=-4038) [2024-04-22 16:23:12.219152] INFO [SHARE] run2 (ob_async_task_queue.cpp:169) [10494][RSAsyncTask1][T0][Y0-000616AB0A0FEC7F-0-0] [lt=6] async task queue stop [2024-04-22 16:23:12.228664] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=125] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:12.244149] INFO acceptfd_handle_first_readable_event (handle-event.c:382) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] recv non-negotiation message, the fd will be dispatched, fd:198, src_addr:172.23.88.130:55316, magic:0x78563412 [2024-04-22 16:23:12.244191] INFO dispatch_accept_fd_to_certain_group (group.c:637) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=33] PNIO dispatch fd to oblistener, fd:198 [2024-04-22 16:23:12.244214] INFO [RPC] read_client_magic (ob_listener.cpp:219) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] read negotiation msg(rcv_byte=19) [2024-04-22 16:23:12.244228] INFO [RPC] read_client_magic (ob_listener.cpp:239) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] read_client_magic, (client_magic=7386785325300370467, index=0) [2024-04-22 16:23:12.244314] INFO [RPC] trace_connection_info (ob_listener.cpp:285) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] oblistener receive connection from(src_addr=172.23.88.130, src_port=55316) [2024-04-22 16:23:12.244341] INFO [RPC] do_one_event (ob_listener.cpp:407) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] dispatch to(client_magic=7386785325300370467, index=0) [2024-04-22 16:23:12.244357] INFO [RPC] connection_redispatch (ob_listener.cpp:261) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] dipatch(conn_fd=198, count=1, index=0, wrfd=97) [2024-04-22 16:23:12.244412] INFO [RPC] connection_redispatch (ob_listener.cpp:267) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] dispatch success!(conn_fd=198, wrfd=97) [2024-04-22 16:23:12.244542] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:456) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=6] new connection established, fd: 198, addr: "172.23.88.130:55316" [2024-04-22 16:23:12.293864] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.293954] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=87] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.293982] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294004] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294033] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294046] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294059] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294072] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294085] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294098] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294112] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294125] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294138] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294159] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294173] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294186] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:12.294981] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] tx gc loop thread is running(MTL_ID()=1) [2024-04-22 16:23:12.295037] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=53] try gc retain ctx [2024-04-22 16:23:12.295140] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:748) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=53] get_max_applied_scn(scn={val:1713774187421454783, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:7, palf_committed_end_lsn_:{lsn:358576}, palf_committed_end_scn_:{val:1713774187421454783, v:0}, last_check_scn_:{val:1713774187421454783, v:0}, max_applied_cb_scn_:{val:1713774187421454783, v:0}}) [2024-04-22 16:23:12.295209] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1023) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=55] get_min_unreplayed_log_info(lsn={lsn:356581}, scn={val:1713774177880657608, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774182065832, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:356581}, next_to_submit_scn_:{val:1713774177880657608, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:356459}, end_lsn:{lsn:356581}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774177880657607, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774177880657607, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:1991013568, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774177880657607, v:0}}}}}) [2024-04-22 16:23:12.295736] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=7] ObArchiveSender is running(thread_index=0) [2024-04-22 16:23:12.297714] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:12.297794] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:12.297832] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774192297701) [2024-04-22 16:23:12.297856] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774192197677, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:12.297920] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=4875803, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.297975] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20] LogLoopThread round_cost_time(us)(round_cost_time=223) [2024-04-22 16:23:12.297971] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=20, Other_Error_Cnt=0, Submit_Succ_Cnt=10, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:12.298027] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1713774187421454783, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1713774192297877}, version_delta=4876572) [2024-04-22 16:23:12.298065] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1957) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] get_fetch_log_dst failed or invalid(ret=0, palf_id=1, self="172.23.88.129:2882", fetch_log_dst="172.23.88.129:2882") [2024-04-22 16:23:12.298105] INFO [PALF] is_pending_log_clear_ (log_state_mgr.cpp:789) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35] follower pending log not clear, need wait(bool_ret=false, palf_id=1, last_slide_end_lsn={lsn:358576}, pending_end_lsn={lsn:358698}) [2024-04-22 16:23:12.298239] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFBED-0-0] [lt=32][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713774192298021}) [2024-04-22 16:23:12.298564] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:253) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=142] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:12.343235] INFO keepalive_check (time_wheel.c:76) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd1aac9b8,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:12.343337] INFO keepalive_check (time_wheel.c:76) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=61] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd33f89e8,dest=-2091378772:2882), old=0, new=3000000 [2024-04-22 16:23:12.343391] INFO keepalive_check (time_wheel.c:71) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=21] PNIO socket dest server in blacklist, it will be destroyed, sock=(ptr=0x7f3cd332c048,dest=-2124933204:2882) [2024-04-22 16:23:12.343492] INFO eloop_handle_sock_event (eloop.c:95) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO sock destroy: sock=0x7f3cd332c048, connection=fd:218:local:172.23.88.129:33114:remote:172.23.88.129:2882, s->mask=0xc [2024-04-22 16:23:12.343626] INFO pktc_sk_delete (pktc_sk_factory.h:84) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO sk_destroy: s=0x7f3cd332c048 io=0x7f3d00d84608 [2024-04-22 16:23:12.343680] WDIAG pkts_sk_consume (handle_io.t.h:57) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] PNIO do_decode fail: 61 [2024-04-22 16:23:12.343723] INFO eloop_handle_sock_event (eloop.c:107) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO sock destroy: sock=0x7f3cd332c9e8, connection=fd:219:local:172.23.88.129:2882:remote:172.23.88.129:33114, err=61 [2024-04-22 16:23:12.343893] INFO pkts_sk_delete (pkts_sk_factory.h:57) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO sk_destroy: s=0x7f3cd332c9e8 io=0x7f3d00c04408 [2024-04-22 16:23:12.393656] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:12.394278] INFO print_current_status (ob_kvcache_hazard_version.cpp:328) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=71] [KVCACHE-HAZARD] hazard version status info: current_version: 0 | min_version= 0 | total_nodes_count: 0 | ?ÿ [2024-04-22 16:23:12.395161] INFO [RPC.OBRPC] try_read_response (ob_net_keepalive.cpp:609) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=4] whitewash, addr: "172.23.88.129:2882", last_write_ts_=1713774192195094, last_read_ts_=1713774192395106 [2024-04-22 16:23:12.397902] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63] get wrs ts(ls_id={id:1}, delta=4975880, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.397982] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=70] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:12.402621] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:12.405320] INFO acceptfd_handle_first_readable_event (handle-event.c:382) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=49] [ussl] recv non-negotiation message, the fd will be dispatched, fd:203, src_addr:172.23.88.131:36226, magic:0x78563412 [2024-04-22 16:23:12.405377] INFO dispatch_accept_fd_to_certain_group (group.c:637) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=45] PNIO dispatch fd to oblistener, fd:203 [2024-04-22 16:23:12.405409] INFO [RPC] read_client_magic (ob_listener.cpp:219) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] read negotiation msg(rcv_byte=19) [2024-04-22 16:23:12.405553] INFO [RPC] read_client_magic (ob_listener.cpp:239) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=141] read_client_magic, (client_magic=7386785325300370467, index=0) [2024-04-22 16:23:12.405590] INFO [RPC] trace_connection_info (ob_listener.cpp:285) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] oblistener receive connection from(src_addr=172.23.88.131, src_port=36226) [2024-04-22 16:23:12.405630] INFO [RPC] do_one_event (ob_listener.cpp:407) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] dispatch to(client_magic=7386785325300370467, index=0) [2024-04-22 16:23:12.405646] INFO [RPC] connection_redispatch (ob_listener.cpp:261) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] dipatch(conn_fd=203, count=1, index=0, wrfd=97) [2024-04-22 16:23:12.405691] INFO [RPC] connection_redispatch (ob_listener.cpp:267) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] dispatch success!(conn_fd=203, wrfd=97) [2024-04-22 16:23:12.405811] INFO [RPC.OBRPC] do_server_loop (ob_net_keepalive.cpp:456) [10599][KeepAliveServer][T0][Y0-0000000000000000-0-0] [lt=33] new connection established, fd: 203, addr: "172.23.88.131:36226" [2024-04-22 16:23:12.445050] INFO keepalive_check (time_wheel.c:76) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd33fe048,dest=-2091378772:2882), old=0, new=3000000 [2024-04-22 16:23:12.445138] INFO keepalive_check (time_wheel.c:76) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=52] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd3372048,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:12.447142] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] sock regist: 0x7f3cfd8726b0 fd=206 [2024-04-22 16:23:12.447210] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=54] [ussl] accept new connection, fd:206, src_addr:172.23.88.130:55318 [2024-04-22 16:23:12.447293] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] auth mothod is NONE, the fd will be dispatched, fd:206, src_addr:172.23.88.130:55318 [2024-04-22 16:23:12.447382] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=78] PNIO dispatch fd to certain group, fd:206, gid:0x100000006 [2024-04-22 16:23:12.447687] INFO pkts_sk_init (pkts_sk_factory.h:23) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO set pkts_sk_t sock_id s=0x7f3cd1aac048, s->id=589821 [2024-04-22 16:23:12.447733] INFO pkts_sk_new (pkts_sk_factory.h:51) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO sk_new: s=0x7f3cd1aac048 [2024-04-22 16:23:12.447808] INFO eloop_regist (eloop.c:47) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO sock regist: 0x7f3cd1aac048 fd=206 [2024-04-22 16:23:12.447859] INFO on_accept (listenfd.c:39) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=28] PNIO accept new connection, ns=0x7f3cd1aac048, fd=fd:206:local:172.23.88.129:2882:remote:172.23.88.130:55318 [2024-04-22 16:23:12.448000] WDIAG listenfd_handle_event (listenfd.c:71) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.448102] INFO [PALF] serialize (log_meta.cpp:185) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=8] LogMeta serialize(*this={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, buf="", buf=0x7f3cc36d2460, pos=779) [2024-04-22 16:23:12.448264] INFO [PALF] serialize_log_meta_ (log_engine.cpp:1528) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=156] serialize_log_meta_ success(ret=0, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_entry={header:{magic:19533, version:1, data_len:4072, data_checksum:3568289559, header_checksum:2107771035}, buf_:0x7f3cc36d2460}, log_meta_entry_header_len=24) [2024-04-22 16:23:12.448370] INFO [PALF] submit_io_task (log_io_worker.cpp:149) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=102] submit flush meta task success(io_task={LogIOTask:{BaseClass:"LogIOTask", palf_id:1, palf_epoch:1, create_task_ts:1713774192448078, push_cb_into_cb_pool_ts:-1, submit_seq_:29}, flush_meta_cb_ctx:{type:"PREPARE_META", proposal_id:8, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}}) [2024-04-22 16:23:12.448462] INFO [PALF] submit_flush_meta_task_ (log_engine.cpp:1287) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=32] submit_flush_meta_task_ success(flush_meta_cb_ctx={type:"PREPARE_META", proposal_id:8, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}) [2024-04-22 16:23:12.448566] INFO [PALF] submit_flush_prepare_meta_task (log_engine.cpp:395) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=102] submit_flush_prepare_meta_task success(ret=0, flush_meta_cb_ctx={type:"PREPARE_META", proposal_id:8, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}, prepare_meta={version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}) [2024-04-22 16:23:12.448597] INFO [PALF] write_prepare_meta_ (log_state_mgr.cpp:548) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=29] write_prepare_meta_ success(prepare_meta={version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, new_leader="172.23.88.130:2882", prepare_meta_={version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}) [2024-04-22 16:23:12.448615] INFO [PALF] handle_prepare_request (log_state_mgr.cpp:468) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=17] LogStateMgr handle_prepare_request success(ret=0, palf_id=1, self_="172.23.88.129:2882", server="172.23.88.130:2882", proposal_id=8, leader_="172.23.88.130:2882", broadcast_leader="172.23.88.130:2882") [2024-04-22 16:23:12.448635] INFO [PALF] clean_log_ (log_sliding_window.cpp:2536) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=15] log_task is valid, check if need clean(ret=0, tmp_log_id=2942, palf_id=1, self="172.23.88.129:2882", log_task={header:{begin_lsn:{lsn:358576}, end_lsn:{lsn:358698}, log_id:2942, min_scn:{val:1713774187494514341, v:0}, max_scn:{val:1713774187494514341, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:358576}, data_checksum:667309872, accum_checksum:2177283792, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1713774192194988, freeze_ts:1713774192194992, submit_ts:1713774192195371, flushed_ts:1713774192214594, gen_to_freeze cost time:4, gen_to_submit cost time:383, submit_to_flush cost time:19223}) [2024-04-22 16:23:12.448672] INFO [PALF] clean_log_ (log_sliding_window.cpp:2544) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=36] find last submit log_task(ret=0, tmp_log_id=2942, palf_id=1, self="172.23.88.129:2882", log_task={header:{begin_lsn:{lsn:358576}, end_lsn:{lsn:358698}, log_id:2942, min_scn:{val:1713774187494514341, v:0}, max_scn:{val:1713774187494514341, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:358576}, data_checksum:667309872, accum_checksum:2177283792, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1713774192194988, freeze_ts:1713774192194992, submit_ts:1713774192195371, flushed_ts:1713774192214594, gen_to_freeze cost time:4, gen_to_submit cost time:383, submit_to_flush cost time:19223}, last_submit_log_id=2942) [2024-04-22 16:23:12.448723] INFO [PALF] truncate (lsn_allocator.cpp:93) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=38] truncate success(lsn={lsn:358698}, log_id=2942, scn={val:1713774187494514341, v:0}) [2024-04-22 16:23:12.448736] INFO [PALF] truncate_lsn_allocator_ (log_sliding_window.cpp:4366) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=12] lsn_allocator_.truncate success(ret=0, palf_id=1, self="172.23.88.129:2882", last_lsn={lsn:358698}, last_log_id=2942, last_scn={val:1713774187494514341, v:0}) [2024-04-22 16:23:12.448750] INFO [PALF] clean_log_ (log_sliding_window.cpp:2576) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=13] clean log finished(ret=0, palf_id=1, self="172.23.88.129:2882", max_log_id=2942, first_empty_log_id=-1, last_submit_log_id=2942, start_log_id=2942, max_log_id=2942, new_last_log_id=2942, new_last_scn={val:1713774187494514341, v:0}, new_last_log_end_lsn={lsn:358698}) [2024-04-22 16:23:12.448769] INFO [PALF] try_update_proposal_id_ (palf_handle_impl.cpp:2960) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=18] try_update_proposal_id_ finished(ret=0, server="172.23.88.130:2882", proposal_id=8) [2024-04-22 16:23:12.448786] WDIAG [PALF] is_prev_log_pid_match (log_sliding_window.cpp:3875) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=9][errcode=-4018] get_prev_log_info_ failed(tmp_ret=-4018, log_id=2951, palf_id=1, self="172.23.88.129:2882", log_id=2951, lsn={lsn:359726}, prev_lsn={lsn:359604}, prev_log_pid=8) [2024-04-22 16:23:12.448804] WDIAG [PALF] receive_log (log_sliding_window.cpp:3199) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1C-0-0] [lt=17][errcode=-4023] new log's proposal_id does not equal to last submit log's, and log_id is not continuous with last submit log, cannot receive(cache) it(ret=-4023, palf_id=1, self="172.23.88.129:2882", log_id=2951, log_proposal_id=8, prev_log_proposal_id=8, is_prev_log_exist=false, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359726}, max_scn:{val:1713774197210958134, v:0}, accumulated_checksum:4063163828, log_id:2951, flag:1}, last_submit_log_id=2942, last_submit_log_pid=7) [2024-04-22 16:23:12.453180] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=6] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=106496, this={dio_aligned_buf:{buf_write_offset:0, buf_padding_size:0, align_size:4096, aligned_buf_size:8192, aligned_used_ts:238, truncate_used_ts:9}, log_block_size:67108864, dir_fd:234, io_fd:236}, aligned_buf_len=4096, aligned_buf="LM", aligned_block_offset=106496, buf_len=4096, total_write_size_=12288, total_write_size_after_dio_=12288, ob_pwrite_used_ts=10335, count=3, each_pwrite_cost=3445) [2024-04-22 16:23:12.453264] INFO [PALF] append_meta (log_storage.cpp:269) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=79] LogStorage append meta success(ret=0, log_block_header_={magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, log_tail_={lsn:106496}, this={log_tail:{lsn:106496}, readable_log_tail:{lsn:106496}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66998272, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}) [2024-04-22 16:23:12.453369] INFO [PALF] handle_io_task_ (log_io_worker.cpp:227) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=78] [PALF STAT IO STAT](do_task_used_ts=13897, do_task_count=4, average_cost_ts=3474, io_queue_size=0) [2024-04-22 16:23:12.453407] INFO [PALF] inner_after_flush_meta (palf_handle_impl.cpp:4174) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=20] inner_after_flush_meta(flush_meta_cb_ctx={type:"PREPARE_META", proposal_id:8, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}) [2024-04-22 16:23:12.453615] INFO [PALF] submit_prepare_meta_resp (log_net_service.cpp:229) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=85] submit_prepare_meta_resp success(ret=0, server="172.23.88.130:2882", msg_proposal_id=8) [2024-04-22 16:23:12.453661] INFO [PALF] submit_prepare_response_ (palf_handle_impl.cpp:4448) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=43] submit_prepare_response success(ret=0, palf_id=1, self="172.23.88.129:2882", server="172.23.88.130:2882", vote_granted=true, accept_proposal_id=7, max_flushed_end_lsn={lsn:358698}, committed_end_lsn={lsn:358576}, accepted_mode_meta={version:1, proposal_id:7, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:12.453699] INFO [PALF] after_flush_prepare_meta_ (palf_handle_impl.cpp:4236) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=36] after_flush_prepare_meta_ success(proposal_id=8) [2024-04-22 16:23:12.454210] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] sock regist: 0x7f3cfd8726b0 fd=207 [2024-04-22 16:23:12.454285] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=61] [ussl] accept new connection, fd:207, src_addr:172.23.88.130:55320 [2024-04-22 16:23:12.454369] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] auth mothod is NONE, the fd will be dispatched, fd:207, src_addr:172.23.88.130:55320 [2024-04-22 16:23:12.454409] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] PNIO dispatch fd to certain group, fd:207, gid:0x100000001 [2024-04-22 16:23:12.454499] INFO pkts_sk_init (pkts_sk_factory.h:23) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO set pkts_sk_t sock_id s=0x7f3d01a76048, s->id=589823 [2024-04-22 16:23:12.454550] INFO pkts_sk_new (pkts_sk_factory.h:51) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=49] PNIO sk_new: s=0x7f3d01a76048 [2024-04-22 16:23:12.454630] INFO eloop_regist (eloop.c:47) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO sock regist: 0x7f3d01a76048 fd=207 [2024-04-22 16:23:12.454720] INFO on_accept (listenfd.c:39) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=59] PNIO accept new connection, ns=0x7f3d01a76048, fd=fd:207:local:172.23.88.129:2882:remote:172.23.88.130:55320 [2024-04-22 16:23:12.454859] WDIAG [PALF] process (log_rpc_proxy.cpp:44) [10459][pnio1][T0][YB42AC175881-000616AB0A3FF24C-0-0] [lt=19][errcode=-4109] there is an rpc error in logservice(rcode={code:-4109, msg:"", warnings:[]}, dst="172.23.88.130:2882", pcode=5381) [2024-04-22 16:23:12.454934] WDIAG listenfd_handle_event (listenfd.c:71) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.497985] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:12.498077] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774192497970) [2024-04-22 16:23:12.498115] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774192297875, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:12.498192] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=5076699, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.498233] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:12.594768] INFO [RPC.OBRPC] check_connect (ob_net_keepalive.cpp:548) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=56] connect ok, fd: 204, conn: "172.23.88.131:2882" [2024-04-22 16:23:12.594983] INFO [RPC.OBRPC] check_connect (ob_net_keepalive.cpp:548) [10600][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=108] connect ok, fd: 205, conn: "172.23.88.130:2882" [2024-04-22 16:23:12.595060] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:12.598129] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] get wrs ts(ls_id={id:1}, delta=5176321, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.598237] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=96] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:12.602779] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:12.630170] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] sock regist: 0x7f3cfd8726b0 fd=208 [2024-04-22 16:23:12.630238] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=54] [ussl] accept new connection, fd:208, src_addr:172.23.88.131:36228 [2024-04-22 16:23:12.630327] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] auth mothod is NONE, the fd will be dispatched, fd:208, src_addr:172.23.88.131:36228 [2024-04-22 16:23:12.630360] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:208, gid:0x100000006 [2024-04-22 16:23:12.630480] INFO pkts_sk_init (pkts_sk_factory.h:23) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO set pkts_sk_t sock_id s=0x7f3cd33f9388, s->id=524287 [2024-04-22 16:23:12.630537] INFO pkts_sk_new (pkts_sk_factory.h:51) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=55] PNIO sk_new: s=0x7f3cd33f9388 [2024-04-22 16:23:12.630616] INFO eloop_regist (eloop.c:47) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO sock regist: 0x7f3cd33f9388 fd=208 [2024-04-22 16:23:12.630690] INFO on_accept (listenfd.c:39) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=51] PNIO accept new connection, ns=0x7f3cd33f9388, fd=fd:208:local:172.23.88.129:2882:remote:172.23.88.131:36228 [2024-04-22 16:23:12.630797] WDIAG listenfd_handle_event (listenfd.c:71) [10464][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.632681] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x7f3cfd8726b0 fd=211 [2024-04-22 16:23:12.632736] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=42] [ussl] accept new connection, fd:211, src_addr:172.23.88.131:36230 [2024-04-22 16:23:12.632815] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] auth mothod is NONE, the fd will be dispatched, fd:211, src_addr:172.23.88.131:36230 [2024-04-22 16:23:12.632842] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] PNIO dispatch fd to certain group, fd:211, gid:0x100000000 [2024-04-22 16:23:12.632992] INFO pkts_sk_init (pkts_sk_factory.h:23) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO set pkts_sk_t sock_id s=0x7f3cd3205358, s->id=589823 [2024-04-22 16:23:12.633042] INFO pkts_sk_new (pkts_sk_factory.h:51) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=48] PNIO sk_new: s=0x7f3cd3205358 [2024-04-22 16:23:12.633120] INFO eloop_regist (eloop.c:47) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO sock regist: 0x7f3cd3205358 fd=211 [2024-04-22 16:23:12.633163] INFO on_accept (listenfd.c:39) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=21] PNIO accept new connection, ns=0x7f3cd3205358, fd=fd:211:local:172.23.88.129:2882:remote:172.23.88.131:36230 [2024-04-22 16:23:12.633289] WDIAG listenfd_handle_event (listenfd.c:71) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.638957] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] sock regist: 0x7f3cfd8726b0 fd=212 [2024-04-22 16:23:12.639030] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=59] [ussl] accept new connection, fd:212, src_addr:172.23.88.131:36232 [2024-04-22 16:23:12.639130] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] auth mothod is NONE, the fd will be dispatched, fd:212, src_addr:172.23.88.131:36232 [2024-04-22 16:23:12.639244] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=104] PNIO dispatch fd to certain group, fd:212, gid:0x100000004 [2024-04-22 16:23:12.639387] INFO pkts_sk_init (pkts_sk_factory.h:23) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO set pkts_sk_t sock_id s=0x7f3cd33fe9e8, s->id=524287 [2024-04-22 16:23:12.639482] INFO pkts_sk_new (pkts_sk_factory.h:51) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=93] PNIO sk_new: s=0x7f3cd33fe9e8 [2024-04-22 16:23:12.639554] INFO eloop_regist (eloop.c:47) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO sock regist: 0x7f3cd33fe9e8 fd=212 [2024-04-22 16:23:12.639648] INFO on_accept (listenfd.c:39) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=66] PNIO accept new connection, ns=0x7f3cd33fe9e8, fd=fd:212:local:172.23.88.129:2882:remote:172.23.88.131:36232 [2024-04-22 16:23:12.639793] WDIAG listenfd_handle_event (listenfd.c:71) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.641591] INFO keepalive_check (time_wheel.c:76) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd24d9358,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:12.645189] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] sock regist: 0x7f3cfd8726b0 fd=213 [2024-04-22 16:23:12.645265] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=59] [ussl] accept new connection, fd:213, src_addr:172.23.88.130:55322 [2024-04-22 16:23:12.645354] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] [ussl] auth mothod is NONE, the fd will be dispatched, fd:213, src_addr:172.23.88.130:55322 [2024-04-22 16:23:12.645396] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=32] PNIO dispatch fd to certain group, fd:213, gid:0x100000005 [2024-04-22 16:23:12.645712] INFO pkts_sk_init (pkts_sk_factory.h:23) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO set pkts_sk_t sock_id s=0x7f3d01a769b8, s->id=524287 [2024-04-22 16:23:12.645803] INFO pkts_sk_new (pkts_sk_factory.h:51) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=89] PNIO sk_new: s=0x7f3d01a769b8 [2024-04-22 16:23:12.645883] INFO eloop_regist (eloop.c:47) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO sock regist: 0x7f3d01a769b8 fd=213 [2024-04-22 16:23:12.645924] INFO on_accept (listenfd.c:39) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO accept new connection, ns=0x7f3d01a769b8, fd=fd:213:local:172.23.88.129:2882:remote:172.23.88.130:55322 [2024-04-22 16:23:12.646046] WDIAG listenfd_handle_event (listenfd.c:71) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.646157] INFO [PALF] can_receive_config_log (log_config_mgr.cpp:2029) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6A-0-0] [lt=31] can_receive_config_log(bool_ret=true, palf_id=1, self="172.23.88.129:2882", leader="172.23.88.130:2882", meta={version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, log_ms_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:12.646523] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2107) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6A-0-0] [lt=186] do_fetch_log_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", dest="172.23.88.130:2882", fetch_log_count=2048, fetch_start_lsn={lsn:358576}, prev_lsn={lsn:358454}, fetch_start_log_id=2942, last_slide_log_id=2941, fetch_log_size=41943040, accepted_mode_pid=7, last_fetch_req_time=1713774192646521, last_fetch_end_lsn={lsn:42301616}, last_fetch_max_log_id=4989, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="MODE_META_BARRIER", this={palf_id:1, self:"172.23.88.129:2882", lsn_allocator:{max_log_id:2942, max_lsn:358698, max_scn:{val:1713774187494514341, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:358698}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:358576}, last_submit_end_lsn:{lsn:358698}, last_submit_log_id:2942, last_submit_log_pid:7, max_flushed_lsn:{lsn:358576}, max_flushed_end_lsn:{lsn:358698}, max_flushed_log_pid:7, committed_end_lsn:{lsn:358576}, last_slide_log_id:2941, last_slide_scn:{val:1713774187421454783, v:0}, last_slide_lsn:{lsn:358454}, last_slide_end_lsn:{lsn:358576}, last_slide_log_pid:7, last_slide_log_accum_checksum:1087531090, last_fetch_end_lsn:{lsn:42301616}, last_fetch_max_log_id:4989, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713774192646521, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"MODE_META_BARRIER", this:0x7f3cd2de4530}) [2024-04-22 16:23:12.646660] INFO [PALF] receive_config_log (palf_handle_impl.cpp:4046) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6A-0-0] [lt=133] pre_check_for_mode_meta don't match, try fetch mode meta(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", msg_proposal_id=8, prev_mode_pid=8, meta={version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}) [2024-04-22 16:23:12.648592] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3617) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=87] receive log with larger proposal_id, maybe need truncate(palf_id=1, self="172.23.88.129:2882", log_id=2942, new_pid=8, header={magic:18258, version:1, group_size:112, proposal_id:8, committed_lsn:{lsn:358576}, max_scn:{val:1713774187421454784, v:0}, accumulated_checksum:3241726231, log_id:2942, flag:1}, log_task={header:{begin_lsn:{lsn:358576}, end_lsn:{lsn:358698}, log_id:2942, min_scn:{val:1713774187494514341, v:0}, max_scn:{val:1713774187494514341, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:358576}, data_checksum:667309872, accum_checksum:2177283792, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1713774192194988, freeze_ts:1713774192194992, submit_ts:1713774192195371, flushed_ts:1713774192214594, gen_to_freeze cost time:4, gen_to_submit cost time:383, submit_to_flush cost time:19223}) [2024-04-22 16:23:12.648766] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=171] no need update log(log_id=2942, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=false, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:112, proposal_id:8, committed_lsn:{lsn:358576}, max_scn:{val:1713774187421454784, v:0}, accumulated_checksum:3241726231, log_id:2942, flag:1}, log_task={header:{begin_lsn:{lsn:358576}, end_lsn:{lsn:358698}, log_id:2942, min_scn:{val:1713774187494514341, v:0}, max_scn:{val:1713774187494514341, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:358576}, data_checksum:667309872, accum_checksum:2177283792, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1713774192194988, freeze_ts:1713774192194992, submit_ts:1713774192195371, flushed_ts:1713774192214594, gen_to_freeze cost time:4, gen_to_submit cost time:383, submit_to_flush cost time:19223}) [2024-04-22 16:23:12.648870] WDIAG [PALF] receive_log (log_sliding_window.cpp:3265) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=100][errcode=-4023] need truncate log(ret=-4023, log_id=2942, palf_id=1, self="172.23.88.129:2882", lsn={lsn:358576}, last_submit_end_lsn={lsn:358698}, last_submit_log_id=2942, log_task={header:{begin_lsn:{lsn:358576}, end_lsn:{lsn:358698}, log_id:2942, min_scn:{val:1713774187494514341, v:0}, max_scn:{val:1713774187494514341, v:0}, data_len:66, proposal_id:7, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:358576}, data_checksum:667309872, accum_checksum:2177283792, is_padding_log:false, is_raw_write:false}, state_map:{val:30}, ref_cnt:0, gen_ts:1713774192194988, freeze_ts:1713774192194992, submit_ts:1713774192195371, flushed_ts:1713774192214594, gen_to_freeze cost time:4, gen_to_submit cost time:383, submit_to_flush cost time:19223}, is_prev_log_exist=true, is_prev_log_match=true, truncate_log_info={truncate_type:"TRUNCATE_LOG", truncate_log_id:2942, truncate_begin_lsn:{lsn:358576}, truncate_log_proposal_id:7}) [2024-04-22 16:23:12.648950] INFO [PALF] receive_log_ (palf_handle_impl.cpp:3066) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=75] begin clean/truncate log(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", self="172.23.88.129:2882", msg_proposal_id=8, lsn={lsn:358576}, buf_len=168, truncate_log_info={truncate_type:"TRUNCATE_LOG", truncate_log_id:2942, truncate_begin_lsn:{lsn:358576}, truncate_log_proposal_id:7}) [2024-04-22 16:23:12.649035] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x7f3cfd8726b0 fd=214 [2024-04-22 16:23:12.649087] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:214, src_addr:172.23.88.130:55324 [2024-04-22 16:23:12.649153] INFO [PALF] truncate (fixed_sliding_window.h:369) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=34] FixedSlidingWindow begin truncate [2024-04-22 16:23:12.649161] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] auth mothod is NONE, the fd will be dispatched, fd:214, src_addr:172.23.88.130:55324 [2024-04-22 16:23:12.649186] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] PNIO dispatch fd to certain group, fd:214, gid:0x100000000 [2024-04-22 16:23:12.649202] INFO [PALF] truncate (fixed_sliding_window.h:371) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=31] FixedSlidingWindow truncate lock success [2024-04-22 16:23:12.649314] INFO pkts_sk_init (pkts_sk_factory.h:23) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=31] PNIO set pkts_sk_t sock_id s=0x7f3cd33ff358, s->id=589821 [2024-04-22 16:23:12.649363] INFO pkts_sk_new (pkts_sk_factory.h:51) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=47] PNIO sk_new: s=0x7f3cd33ff358 [2024-04-22 16:23:12.649469] INFO eloop_regist (eloop.c:47) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO sock regist: 0x7f3cd33ff358 fd=214 [2024-04-22 16:23:12.649568] INFO on_accept (listenfd.c:39) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=65] PNIO accept new connection, ns=0x7f3cd33ff358, fd=fd:214:local:172.23.88.129:2882:remote:172.23.88.130:55324 [2024-04-22 16:23:12.649717] WDIAG listenfd_handle_event (listenfd.c:71) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.650305] INFO [PALF] truncate (fixed_sliding_window.h:387) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=12] FixedSlidingWindow truncate success(ret=0, t_id=2942, begin_sn_=2942, end_sn_=4990) [2024-04-22 16:23:12.650596] INFO [PALF] submit_truncate_log_task (log_engine.cpp:505) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=53] submit_truncate_log_task success(ret=0, truncate_log_cb_ctx={lsn:{lsn:358576}}) [2024-04-22 16:23:12.650685] INFO [PALF] truncate (lsn_allocator.cpp:93) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=70] truncate success(lsn={lsn:358576}, log_id=2941, scn={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.650724] INFO [PALF] truncate_lsn_allocator_ (log_sliding_window.cpp:4366) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=36] lsn_allocator_.truncate success(ret=0, palf_id=1, self="172.23.88.129:2882", last_lsn={lsn:358576}, last_log_id=2941, last_scn={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.650779] INFO [PALF] set_accum_checksum (log_checksum.cpp:131) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=23] set_accum_checksum(palf_id=1, accum_checksum=1087531090) [2024-04-22 16:23:12.650814] INFO [PALF] truncate (log_sliding_window.cpp:3091) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=33] truncate last_submit_log_info_ and accum_checksum(palf_id=1, self="172.23.88.129:2882", truncate_log_info={truncate_type:"TRUNCATE_LOG", truncate_log_id:2942, truncate_begin_lsn:{lsn:358576}, truncate_log_proposal_id:7}, last_submit_log_id=2942, prev_accum_checksum=1087531090) [2024-04-22 16:23:12.650842] INFO [PALF] truncate_max_flushed_log_info_ (log_sliding_window.cpp:1809) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=26] truncate_max_flushed_log_info_ success(palf_id=1, self="172.23.88.129:2882", lsn={lsn:358454}, end_lsn={lsn:358576}, proposal_id=7) [2024-04-22 16:23:12.650860] INFO [PALF] truncate (log_group_buffer.cpp:467) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=17] LogGroupBuffer truncate success(curr_start_lsn={lsn:0}, old_reuse_lsn={lsn:358698}, old_readable_begin_lsn={lsn:0}, readable_begin_lsn={lsn:358576}, new_lsn={lsn:358576}) [2024-04-22 16:23:12.650879] INFO [PALF] truncate (log_sliding_window.cpp:3102) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=18] truncate max_flushed_log_info_(palf_id=1, self="172.23.88.129:2882", truncate_log_info={truncate_type:"TRUNCATE_LOG", truncate_log_id:2942, truncate_begin_lsn:{lsn:358576}, truncate_log_proposal_id:7}, log_end_lsn={lsn:358698}, old flushed_end_lsn={lsn:358698}) [2024-04-22 16:23:12.650901] INFO [PALF] truncate (log_sliding_window.cpp:3105) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=19] truncate success(ret=0, palf_id=1, self="172.23.88.129:2882", truncate_log_info={truncate_type:"TRUNCATE_LOG", truncate_log_id:2942, truncate_begin_lsn:{lsn:358576}, truncate_log_proposal_id:7}, max_log_id=2941, log_begin_lsn={lsn:358576}, expected_prev_lsn={lsn:358454}, expected_prev_log_pid=7, prev_accum_checksum=1087531090) [2024-04-22 16:23:12.650927] INFO [PALF] truncate (log_state_mgr.cpp:748) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=24] state_mgr truncate finished(ret=0, palf_id=1, pending_end_lsn={lsn:358576}, truncate_begin_lsn={lsn:358576}) [2024-04-22 16:23:12.650962] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=27][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:358576}, buf=0x7f3cbce084ff, buf_len=168) [2024-04-22 16:23:12.650990] WDIAG [PALF] receive_log_ (palf_handle_impl.cpp:3099) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=27][errcode=-4109] sw_ receive_log failed(ret=-4109, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", msg_proposal_id=8, lsn={lsn:358576}) [2024-04-22 16:23:12.651011] WDIAG [PALF] receive_batch_log (palf_handle_impl.cpp:3156) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=19][errcode=-4109] receive_log failed(ret=-4109, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, iterator={iterator_impl:{buf_:0x7f3cbce084ff, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1272, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:359848}, read_buf:{buf_len_:1272, buf_:0x7f3cbce084ff}, block_size:67104768, log_storage_:0x7f3cc36d37d0}, IteratorStorageType::"MemoryIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:168, prev_entry_scn:{val:1713774187421454784, v:0}, curr_entry:{LogGroupEntryHeader:{magic:18258, version:1, group_size:112, proposal_id:8, committed_lsn:{lsn:358576}, max_scn:{val:1713774187421454784, v:0}, accumulated_checksum:3241726231, log_id:2942, flag:1}}, init_mode_version:0, accumulate_checksum:3241726231, curr_entry_is_padding:0, padding_entry_size:112, padding_entry_scn:{val:1713774187421454784, v:0}}}, server="172.23.88.130:2882", FETCH_LOG_RESP=1, msg_proposal_id=8, prev_lsn_each_round={lsn:358454}, prev_log_proposal_id_each_round=7, curr_lsn_each_round={lsn:358576}, buf_each_round=0x7f3cbce084ff, buf_len_each_round=168) [2024-04-22 16:23:12.651091] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=71][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:358744}, buf=0x7f3cbce085a7, buf_len=122) [2024-04-22 16:23:12.651112] WDIAG [PALF] receive_log_ (palf_handle_impl.cpp:3056) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=20][errcode=-4109] sw_ receive_log failed(ret=-4109, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", msg_proposal_id=8, lsn={lsn:358744}) [2024-04-22 16:23:12.651131] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=17][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:358866}, buf=0x7f3cbce08621, buf_len=122) [2024-04-22 16:23:12.651152] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=18][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:358988}, buf=0x7f3cbce0869b, buf_len=122) [2024-04-22 16:23:12.651221] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=62][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:359110}, buf=0x7f3cbce08715, buf_len=128) [2024-04-22 16:23:12.651243] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=19][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:359238}, buf=0x7f3cbce08795, buf_len=122) [2024-04-22 16:23:12.651264] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=18][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:359360}, buf=0x7f3cbce0880f, buf_len=122) [2024-04-22 16:23:12.651284] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=18][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:359482}, buf=0x7f3cbce08889, buf_len=122) [2024-04-22 16:23:12.651306] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=19][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:359604}, buf=0x7f3cbce08903, buf_len=122) [2024-04-22 16:23:12.651326] WDIAG [PALF] receive_log (log_sliding_window.cpp:3147) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F17-0-0] [lt=18][errcode=-4109] sw is truncatiing/rebuilding now, cannot receive log(ret=-4109, palf_id=1, self="172.23.88.129:2882", is_truncating=true, last_truncate_lsn={lsn:358576}, is_rebuilding=false, src_server="172.23.88.130:2882", lsn={lsn:359726}, buf=0x7f3cbce0897d, buf_len=122) [2024-04-22 16:23:12.651521] INFO [PALF] serialize (log_meta.cpp:185) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F18-0-0] [lt=18] LogMeta serialize(*this={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, buf="", buf=0x7f3cc36d2570, pos=779) [2024-04-22 16:23:12.651670] INFO [PALF] serialize_log_meta_ (log_engine.cpp:1528) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F18-0-0] [lt=142] serialize_log_meta_ success(ret=0, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_entry={header:{magic:19533, version:1, data_len:4072, data_checksum:2112875853, header_checksum:379538474}, buf_:0x7f3cc36d2570}, log_meta_entry_header_len=24) [2024-04-22 16:23:12.651826] INFO [PALF] submit_io_task (log_io_worker.cpp:149) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F18-0-0] [lt=151] submit flush meta task success(io_task={LogIOTask:{BaseClass:"LogIOTask", palf_id:1, palf_epoch:1, create_task_ts:1713774192651499, push_cb_into_cb_pool_ts:-1, submit_seq_:30}, flush_meta_cb_ctx:{type:"MODE_META", proposal_id:8, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}}) [2024-04-22 16:23:12.651866] INFO [PALF] submit_flush_meta_task_ (log_engine.cpp:1287) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F18-0-0] [lt=38] submit_flush_meta_task_ success(flush_meta_cb_ctx={type:"MODE_META", proposal_id:8, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}) [2024-04-22 16:23:12.651980] INFO [PALF] submit_flush_mode_meta_task (log_engine.cpp:437) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F18-0-0] [lt=112] submit_flush_mode_meta_task success(ret=0, flush_meta_cb_ctx={type:"MODE_META", proposal_id:8, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:12.652013] INFO [PALF] receive_mode_meta (palf_handle_impl.cpp:2793) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F18-0-0] [lt=31] receive_mode_meta finish(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", proposal_id=8, is_applied_mode_meta=true, mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:12.652042] INFO [PALF] handle_request (log_request_handler.cpp:335) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F18-0-0] [lt=27] receive_mode_meta success(ret=0, palf_id=1, server="172.23.88.130:2882", req={msg_proposal_id:8, meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:true}, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:1}}) [2024-04-22 16:23:12.655491] INFO [PALF] inner_pread_ (log_storage.cpp:971) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT READ LOG INFO FROM DISK](this={log_tail:{lsn:358698}, readable_log_tail:{lsn:358698}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/log", dir_fd:235, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66746070, block_header_serialize_buf_:0x7f3cd2dec138, flashback_version:0}, accum_read_io_count=1, accum_read_log_size=122, avg_pread_cost=4792) [2024-04-22 16:23:12.655644] INFO [PALF] close (log_block_handler.cpp:234) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=69] LogFileHandler close success(ret=0, this={dio_aligned_buf:{buf_write_offset:2346, buf_padding_size:1750, align_size:4096, aligned_buf_size:2125824, aligned_used_ts:5842, truncate_used_ts:2146}, log_block_size:67108864, dir_fd:235, io_fd:-1}) [2024-04-22 16:23:12.666609] INFO [PALF] open (log_block_handler.cpp:223) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] open block success(ret=0, this={dio_aligned_buf:{buf_write_offset:0, buf_padding_size:0, align_size:4096, aligned_buf_size:2125824, aligned_used_ts:5842, truncate_used_ts:2146}, log_block_size:67108864, dir_fd:235, io_fd:215}) [2024-04-22 16:23:12.671074] INFO [PALF] inner_truncate_ (log_block_handler.cpp:347) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] inner_truncate_ success(ret=0, offset=362672) [2024-04-22 16:23:12.671149] INFO [PALF] truncate (log_block_handler.cpp:245) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=72] LogBlockHandler truncate success(ret=0, offset=362672, this={dio_aligned_buf:{buf_write_offset:0, buf_padding_size:0, align_size:4096, aligned_buf_size:2125824, aligned_used_ts:5842, truncate_used_ts:2146}, log_block_size:67108864, dir_fd:235, io_fd:215}) [2024-04-22 16:23:12.672427] INFO [PALF] check_after_truncate_ (log_block_mgr.cpp:407) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=42] check_after_truncate_ success(this={log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/log", dir_fd:235, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, block_path="0", offset=362672) [2024-04-22 16:23:12.673214] INFO [PALF] inner_load_data_ (log_block_handler.cpp:378) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=79] inner_load_data_ success(ret=0, offset=362672, read_count=2224, dio_aligned_buf_={buf_write_offset:2224, buf_padding_size:1872, align_size:4096, aligned_buf_size:2125824, aligned_used_ts:5845, truncate_used_ts:2146}, aligned_read_count=4096) [2024-04-22 16:23:12.673284] INFO [PALF] load_data (log_block_handler.cpp:256) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=66] LogBlockHandler load_data success(ret=0, offset=362672, this={dio_aligned_buf:{buf_write_offset:2224, buf_padding_size:1872, align_size:4096, aligned_buf_size:2125824, aligned_used_ts:5845, truncate_used_ts:2146}, log_block_size:67108864, dir_fd:235, io_fd:215}) [2024-04-22 16:23:12.673318] INFO [PALF] do_truncate_ (log_block_mgr.cpp:365) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] do_truncate_ success(ret=0, block_id=0, offset=362672, min_block_id_=0, max_block_id_=1) [2024-04-22 16:23:12.673337] INFO [PALF] truncate (log_block_mgr.cpp:200) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=18] truncate success(ret=0, block_id=0, offset=362672, min_block_id_=0, max_block_id_=1) [2024-04-22 16:23:12.673371] INFO [PALF] inner_truncate_ (log_storage.cpp:359) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=13] inner_truncate_ success(ret=0, lsn={lsn:358576}, this={log_tail:{lsn:358576}, readable_log_tail:{lsn:358576}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/log", dir_fd:235, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66746192, block_header_serialize_buf_:0x7f3cd2dec138, flashback_version:0}) [2024-04-22 16:23:12.673481] INFO [PALF] truncate (log_storage.cpp:334) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=108] truncate success(ret=0, lsn={lsn:358576}, this={log_tail:{lsn:358576}, readable_log_tail:{lsn:358576}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/log", dir_fd:235, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66746192, block_header_serialize_buf_:0x7f3cd2dec138, flashback_version:0}) [2024-04-22 16:23:12.673557] INFO [PALF] truncate (log_engine.cpp:702) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=54] truncate success(lsn={lsn:358576}) [2024-04-22 16:23:12.673583] INFO [PALF] inner_truncate_log (palf_handle_impl.cpp:2201) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] PalfHandleImpl inner_truncate_log success(lsn={lsn:358576}, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}) [2024-04-22 16:23:12.673639] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1, this=0x7f3ce8f43380, extra_info="log_io_worker_num:1, cb_thread_pool_tg_id:296") [2024-04-22 16:23:12.673694] INFO [PALF] after_truncate (log_sliding_window.cpp:1406) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=12] after_truncate begin(palf_id=1, self="172.23.88.129:2882", truncate_cb_ctx={lsn:{lsn:358576}}) [2024-04-22 16:23:12.673772] INFO [PALF] truncate (log_group_buffer.cpp:467) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=74] LogGroupBuffer truncate success(curr_start_lsn={lsn:0}, old_reuse_lsn={lsn:358576}, old_readable_begin_lsn={lsn:358576}, readable_begin_lsn={lsn:358576}, new_lsn={lsn:358576}) [2024-04-22 16:23:12.673800] INFO [PALF] after_truncate (log_sliding_window.cpp:1424) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] after_truncate success(palf_id=1, self="172.23.88.129:2882", truncate_cb_ctx={lsn:{lsn:358576}}) [2024-04-22 16:23:12.673837] INFO [PALF] inner_after_truncate_log (palf_handle_impl.cpp:4312) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=36] after_truncate_log success(ret=0, self="172.23.88.129:2882", palf_id=1) [2024-04-22 16:23:12.678630] INFO [PALF] append_meta (log_storage.cpp:269) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] LogStorage append meta success(ret=0, log_block_header_={magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, log_tail_={lsn:110592}, this={log_tail:{lsn:110592}, readable_log_tail:{lsn:110592}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66994176, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}) [2024-04-22 16:23:12.678745] INFO [PALF] inner_after_flush_meta (palf_handle_impl.cpp:4174) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] inner_after_flush_meta(flush_meta_cb_ctx={type:"MODE_META", proposal_id:8, config_version:{proposal_id:9223372036854775807, config_seq:-1}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}) [2024-04-22 16:23:12.678831] INFO [PALF] after_flush_mode_meta (log_mode_mgr.cpp:771) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=65] after_flush_mode_meta success(ret=0, palf_id=1, self="172.23.88.129:2882", is_applied_mode_meta=true, mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, applied_mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:12.678980] INFO [PALF] after_flush_mode_meta_ (palf_handle_impl.cpp:4300) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=41] submit_change_mode_meta_resp success(ret=0, self="172.23.88.129:2882", leader="172.23.88.130:2882", proposal_id=8) [2024-04-22 16:23:12.691462] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [10825][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=66] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-22 16:23:12.696685] INFO [CLOG] handle_sync_request (log_request_handler.cpp:513) [11430][T1_L0_G1][T1][YB42AC175881-000616AB0B4FEC0F-0-0] [lt=66] i am not leader(ret=-4038, palf_id=1, req={get_type:1}, role=2, is_pending_state=true) [2024-04-22 16:23:12.697059] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] execute rpc fail(addr="172.23.88.129:2882", pcode=5407, ret=-4038, timeout=2000000) [2024-04-22 16:23:12.697123] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4038] [2024-04-22 16:23:12.697161] WDIAG [CLOG] post_sync_request_to_server_ (log_net_service.h:347) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] ObLogRpc post_sync_request failed(ret=-4038, palf_id=1, req={get_type:1}, server="172.23.88.129:2882") [2024-04-22 16:23:12.697185] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5195) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] get_palf_max_scn failed(ret=-4038, palf_id=1) [2024-04-22 16:23:12.697236] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5155) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] get_palf_max_scn failed(ret=-4038, self="172.23.88.129:2882", palf_id=1) [2024-04-22 16:23:12.697295] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5173) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=56] is_in_sync(ret=-4038, palf_id=1, is_in_sync=false, is_use_cache=false, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.698020] WDIAG [STORAGE.TRANS] get_number (ob_id_service.cpp:355) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] id service is not leader(ret=-4038, service_type=0) [2024-04-22 16:23:12.698056] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:389) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=34] get number(ret=-4038, service_type_=0, range=1, base_id=1713774192698016564, start_id=0, end_id=94103603484063) [2024-04-22 16:23:12.698079] WDIAG [STORAGE.TRANS] handle_local_request_ (ob_timestamp_service.cpp:191) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] get timestamp failed(ret=-4038, ret="OB_NOT_MASTER") [2024-04-22 16:23:12.698098] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.23.88.129:2882", msg={tenant_id:1, srr:[mts=1713774192698010], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:12.698120] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.23.88.129:2882", msg={tenant_id:1, srr:[mts=1713774192698010], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:12.698150] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774185662216], gts:1713774187494514341, latest_srr:[mts=1713774192698010]}) [2024-04-22 16:23:12.698175] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:175) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-04-22 16:23:12.698184] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-04-22 16:23:12.698193] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:12.698302] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774192698181) [2024-04-22 16:23:12.698359] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774192498139, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:12.698397] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:12.698506] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=5276223, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.698500] INFO [PALF] follower_pending_need_switch_ (log_state_mgr.cpp:802) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22] follower_pending_need_switch_(state_changed=true, palf_id=1) [2024-04-22 16:23:12.698509] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:181) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=6] dummy location not credible, need refresh [2024-04-22 16:23:12.698549] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:232) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=37] waiting for refresh locatition cache for target_monitor(tenant_id_=1, refresh_ctrl=13) [2024-04-22 16:23:12.698558] INFO [PALF] follower_pending_need_switch_ (log_state_mgr.cpp:802) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=39] follower_pending_need_switch_(state_changed=true, palf_id=1) [2024-04-22 16:23:12.698577] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] refresh dummy location cache(ret=0) [2024-04-22 16:23:12.698559] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:12.698589] INFO [CLOG] insert (ob_role_change_service.cpp:98) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23] insert event into set success(ret=0, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, free_idx=0) [2024-04-22 16:23:12.698616] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED36-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.698651] INFO [CLOG] submit_role_change_event_ (ob_role_change_service.cpp:271) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=28] submit_role_change_event_ success(ret=0, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:12.698661] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED36-0-0] [lt=28] table not exist(tenant_id=1, database_id=201001, table_name=__all_service_epoch, table_name.ptr()="data_size:19, data:5F5F616C6C5F736572766963655F65706F6368", ret=-5019) [2024-04-22 16:23:12.698673] INFO [CLOG] on_role_change (ob_role_change_service.cpp:243) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21] on_role_change success(ret=0, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:12.698688] INFO [PALF] to_follower_active_ (log_state_mgr.cpp:571) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] to_follower_active_ finished!(ret=0, palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:12.698687] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10804][T1_HBService][T1][YB42AC175881-000616AB094FED36-0-0] [lt=23][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:12.698696] INFO [CLOG] handle (ob_role_change_service.cpp:217) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=6] begin handle_role_change_event_(sequence:=1713774192698691, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:12.698770] INFO [OCCAM] ObOccamTimerTaskRAIIHandle (ob_occam_timer.h:363) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15] task handle constructed(*this={this:0x7f3ce7dcf510, task:NULL, is_inited:false, is_running:false}) [2024-04-22 16:23:12.698798] INFO [CLOG] remove (ob_role_change_service.cpp:117) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=95] remove slog from set success(ret=0, delete_idx=0, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:12.698876] INFO [SHARE] add_event_with_retry (ob_event_history_table_operator.h:374) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] add_event_with_retry success(ret=0, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774192698717), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER PENDING', 'CURRENT', 'FOLLOWER ACTIVE', '', 0, '', 0, 'leader:"172.23.88.130:2882", pending_end_lsn:{lsn:358576}', '172.23.88.129', 2882)) [2024-04-22 16:23:12.698911] INFO [PALF] pending_to_follower_active_ (log_state_mgr.cpp:588) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] [PALF_EVENT] follower_pending_to_follower_active(palf_id=1, ret=0, self="172.23.88.129:2882", leader="172.23.88.130:2882", pending_end_lsn={lsn:358576}) [2024-04-22 16:23:12.698972] WDIAG [PALF] need_update_leader_ (log_state_mgr.cpp:958) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=54][errcode=0] leader or epoch has changed, need update(bool_ret=true, new_leader="0.0.0.0:0", palf_id=1, new_leader_epoch=0, leader_="172.23.88.130:2882", leader_epoch_=-1, self="172.23.88.129:2882") [2024-04-22 16:23:12.699003] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=28] switch_state(ret=0, palf_id=1, old_role=2, role=2, old_state=4, state=2, old_leader="172.23.88.130:2882", leader="172.23.88.130:2882", old_leader_epoch=-1, leader_epoch=-1, need_next_loop=true, diff=445) [2024-04-22 16:23:12.699048] INFO [PALF] set_leader_and_epoch_ (log_state_mgr.cpp:1000) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=41] set_leader_and_epoch_(palf_id=1, self="172.23.88.129:2882", leader_="0.0.0.0:0", leader_epoch_=0, new_leader="0.0.0.0:0") [2024-04-22 16:23:12.699066] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18] switch_state(ret=0, palf_id=1, old_role=2, role=2, old_state=2, state=2, old_leader="172.23.88.130:2882", leader="0.0.0.0:0", old_leader_epoch=-1, leader_epoch=0, need_next_loop=true, diff=21) [2024-04-22 16:23:12.699088] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19] switch_state(ret=0, palf_id=1, old_role=2, role=2, old_state=2, state=2, old_leader="0.0.0.0:0", leader="0.0.0.0:0", old_leader_epoch=0, leader_epoch=0, need_next_loop=false, diff=1) [2024-04-22 16:23:12.699176] INFO [CLOG] handle_role_change_event_ (ob_role_change_service.cpp:345) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=43] begin log handler role change(curr_access_mode=1, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:1, proposal_id:7, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:12.699496] INFO [CLOG] switch_to_follower_ (ob_log_apply_service.cpp:607) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=224] switch_to_follower submit_task_to_apply_service_ success(this={ls_id_:{id:1}, role_:2, proposal_id_:7, palf_committed_end_lsn_:{lsn:358576}, palf_committed_end_scn_:{val:1713774187421454783, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713774187421454783, v:0}}, ret=0) [2024-04-22 16:23:12.699553] INFO [CLOG] switch_to_follower (ob_log_apply_service.cpp:1252) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=54] apply service switch_to_follower success(id={id:1}) [2024-04-22 16:23:12.699625] INFO [CLOG] try_handle_cb_queue (ob_log_apply_service.cpp:504) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=10] cb on_failure(lsn={lsn:358632}, scn={val:1713774187494514341, v:0}, link->next_=0x7f3cd2df6638, cb_queue={ObApplyServiceQueueTask:{type_:2}, total_submit_cb_cnt_:34, last_check_submit_cb_cnt_:33, total_apply_cb_cnt_:33, idx_:5}, this={ls_id_:{id:1}, role_:2, proposal_id_:7, palf_committed_end_lsn_:{lsn:358576}, palf_committed_end_scn_:{val:1713774187421454783, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713774187421454783, v:0}}) [2024-04-22 16:23:12.699774] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=142] [APPLY STAT CB APPEND COST TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=167, this=0x7f3cd2df8280, extra_info="") [2024-04-22 16:23:12.699801] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=26] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=36830, this=0x7f3cd2df84b0, extra_info="") [2024-04-22 16:23:12.699816] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=14] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=0, this=0x7f3cd2df86e0, extra_info="") [2024-04-22 16:23:12.699829] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=12] [APPLY STAT CB EXECUTE TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=-493, this=0x7f3cd2df8910, extra_info="") [2024-04-22 16:23:12.699841] INFO [LIB] stat (utility.h:1165) [10815][T1_ApplySrv0][T1][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB TOTAL TIME](cur_stat_count=16, stat_interval=5000000, avg (count/cost)=36504, this=0x7f3cd2df8b40, extra_info="") [2024-04-22 16:23:12.700806] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC64-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:12.700887] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC64-0-0] [lt=58] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:12.700928] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC64-0-0] [lt=36][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:12.704699] WDIAG [CLOG] wait_apply_service_apply_done_ (ob_role_change_service.cpp:908) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] wait apply done return false, need retry(ls_id={id:1}, is_done=false, end_lsn={lsn:18446744073709551615}, start_ts=1713774192699578) [2024-04-22 16:23:12.704825] INFO [STORAGE.TRANS] switch_state (ob_trans_ctx_mgr_v4.cpp:365) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=67] ObLSTxCtxMgr switch state success(ls_id=1, L_WORKING ~> F_WORKING, op=LEADER_REVOKE) [2024-04-22 16:23:12.707543] INFO [STORAGE.TRANS] switch_to_follower_forcedly (ob_trans_ctx_mgr_v4.cpp:787) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=22] [LsTxCtxMgr Role Change] switch_to_follower_forcedly(ret=0, this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}) [2024-04-22 16:23:12.707636] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=90] leader to follower forcedly, current role change handler is(cursor=1, name="TRANS_SERVICE") [2024-04-22 16:23:12.707663] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=23] leader to follower forcedly, current role change handler is(cursor=3, name="STORAGE_SCHEMA") [2024-04-22 16:23:12.707678] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=13] leader to follower forcedly, current role change handler is(cursor=4, name="TABLET_SEQ_SYNC") [2024-04-22 16:23:12.707690] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] leader to follower forcedly, current role change handler is(cursor=5, name="DDL") [2024-04-22 16:23:12.707701] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=9] leader to follower forcedly, current role change handler is(cursor=6, name="KEEP_ALIVE") [2024-04-22 16:23:12.707738] INFO [STORAGE.TRANS] switch_to_follower_forcedly (ob_timestamp_service.cpp:222) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=33] ObTimestampService switch to follower forcedly success(type=1, service_type=0) [2024-04-22 16:23:12.707752] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=13] leader to follower forcedly, current role change handler is(cursor=7, name="TIMESTAMP") [2024-04-22 16:23:12.707764] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] leader to follower forcedly, current role change handler is(cursor=8, name="TRANS_ID") [2024-04-22 16:23:12.707778] INFO [CLOG] switch_to_follower_forcedly (ob_garbage_collector.cpp:647) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] gc_handler switch_to_follower_forcedly [2024-04-22 16:23:12.707790] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=12] leader to follower forcedly, current role change handler is(cursor=9, name="GC_LS") [2024-04-22 16:23:12.707986] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=158] leader to follower forcedly, current role change handler is(cursor=10, name="MAJOR_FREEZE") [2024-04-22 16:23:12.708015] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=25] leader to follower forcedly, current role change handler is(cursor=14, name="GAIS") [2024-04-22 16:23:12.708029] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=12] leader to follower forcedly, current role change handler is(cursor=15, name="DAS_ID") [2024-04-22 16:23:12.708093] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=33] leader to follower forcedly, current role change handler is(cursor=16, name="RESTORE_SERVICE") [2024-04-22 16:23:12.708106] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=11] leader to follower forcedly, current role change handler is(cursor=17, name="RESERVED_SNAPSHOT") [2024-04-22 16:23:12.708117] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] leader to follower forcedly, current role change handler is(cursor=18, name="MEDIUM_COMPACTION") [2024-04-22 16:23:12.708130] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] leader to follower forcedly, current role change handler is(cursor=23, name="NET_ENDPOINT_EGRESS") [2024-04-22 16:23:12.708165] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=11] leader to follower forcedly, current role change handler is(cursor=24, name="HEARTBEAT_SERVICE") [2024-04-22 16:23:12.708188] INFO [STORAGE.DUP_TABLE] leader_revoke_ (ob_dup_table_util.cpp:1417) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=9] Leader Revoke(ret=0, is_forcedly=true, this={ls_id_:{id:1}, ls_state_helper_:{module_name:"DupTableLSHandler", cur_state_:{role_state_:1, offline_state_:21, stop_state_:31}}, dup_ls_ckpt_:{dup_ls_meta_:{ls_id_:{id:1}, lease_log_applied_scn_:{val:18446744073709551615, v:3}, readable_tablets_base_scn_:{val:18446744073709551615, v:3}, readable_tablets_min_base_applied_scn_:{val:18446744073709551615, v:3}, lease_item_array_.count():0}, lease_log_rec_scn_:{val:18446744073709551615, v:3}, start_replay_scn_:{val:18446744073709551615, v:3}, readable_ckpt_base_scn_is_accurate_:true}, lease_mgr_ptr_:null, tablets_mgr_ptr_:null, ts_sync_mgr_ptr_:null, log_operator_:null, interface_stat_:{dup_table_follower_read_succ_cnt_:0, dup_table_follower_read_tablet_not_exist_cnt_:0, dup_table_follower_read_tablet_not_ready_cnt_:0, dup_table_follower_read_lease_expired_cnt_:0, dup_table_redo_sync_succ_cnt_:0, dup_table_redo_sync_fail_cnt_:0, dup_table_max_applying_scn_:{val:0, v:0}, dup_table_log_entry_cnt_:0, dup_table_log_entry_total_size_:0, dup_table_log_replay_total_time_:0, dup_table_log_deser_total_time_:0, dup_table_lease_log_sync_total_time_:0, dup_table_tablet_log_sync_total_time_:0}}) [2024-04-22 16:23:12.708288] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=97] leader to follower forcedly, current role change handler is(cursor=26, name="DUP_TABLE") [2024-04-22 16:23:12.708328] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=26] leader to follower forcedly, current role change handler is(cursor=27, name="OBJ_LOCK_GARBAGE_COLLECT_SERVICE") [2024-04-22 16:23:12.708431] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=11] leader to follower forcedly, current role change handler is(cursor=31, name="BACKUP_TASK_SCHEDULER") [2024-04-22 16:23:12.708566] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] leader to follower forcedly, current role change handler is(cursor=32, name="BACKUP_DATA_SERVICE") [2024-04-22 16:23:12.708637] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=17] leader to follower forcedly, current role change handler is(cursor=33, name="BACKUP_CLEAN_SERVICE") [2024-04-22 16:23:12.708706] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=9] leader to follower forcedly, current role change handler is(cursor=34, name="BACKUP_ARCHIVE_SERVICE") [2024-04-22 16:23:12.708719] INFO [STORAGE] switch_to_follower_forcedly (ob_transfer_handler.cpp:281) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] [TRANSFER]switch to follower finish [2024-04-22 16:23:12.708731] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=11] leader to follower forcedly, current role change handler is(cursor=35, name="TRANSFER_HANDLER") [2024-04-22 16:23:12.708766] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=12] leader to follower forcedly, current role change handler is(cursor=36, name="COMMON_LS_SERVICE") [2024-04-22 16:23:12.708807] WDIAG [STORAGE.TRANS] switch_state (ob_trans_ctx_mgr_v4.cpp:368) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4109] ObLSTxCtxMgr switch state failed(ret=-4109, ls_id=1, state=F_WORKING, op=UNBLOCK_NORMAL) [2024-04-22 16:23:12.708831] WDIAG [STORAGE.TRANS] unblock_normal (ob_trans_ctx_mgr_v4.cpp:1102) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4109] switch state error(ret=-4109, ret="OB_STATE_NOT_MATCH", manager={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}) [2024-04-22 16:23:12.708901] INFO [STORAGE.TRANS] unblock_normal (ob_trans_ctx_mgr_v4.cpp:1104) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=69] unblock ls normally(ret=-4109, manager={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}) [2024-04-22 16:23:12.708935] WDIAG [STORAGE.TRANS] unblock_normal (ob_ls_tx_service.cpp:843) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4109] ls tx service unblock normal failed(ret=-4109, ls_id={id:1}) [2024-04-22 16:23:12.708949] INFO [STORAGE] switch_to_follower_forcedly (ob_ls_block_tx_service.cpp:65) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=13] [BLOCK_TX]switch to follower finish [2024-04-22 16:23:12.708960] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] leader to follower forcedly, current role change handler is(cursor=37, name="BLOCK_TX_SERVICE") [2024-04-22 16:23:12.708991] INFO cancel_task (ob_timer.cpp:276) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=10] cancel task(this=0x7f3cfba83d10, thread_id=10679, wakeup_time_=4538015310, tasks_num_=0, task={ObTimerTask:tasktype:N9oceanbase5share22WorkloadRepositoryTaskE, snapshot_interval:60, tg_id:290, timeout_ts:0, is_running_task:false}) [2024-04-22 16:23:12.709028] INFO [WR] inner_switch_to_follower (ob_wr_service.cpp:110) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=34] stop to execute workload repository snapshot timer(this={ObIRoleChangeSubHandler:{ObIRoleChangeSubHandler:"Dummy"}, is_inited:true, wr_timer_task:{ObTimerTask:tasktype:N9oceanbase5share22WorkloadRepositoryTaskE, snapshot_interval:60, tg_id:290, timeout_ts:0, is_running_task:false}}) [2024-04-22 16:23:12.709053] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=24] leader to follower forcedly, current role change handler is(cursor=38, name="WORKLOAD_REPOSITORY_SERVICE") [2024-04-22 16:23:12.709090] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=12] leader to follower forcedly, current role change handler is(cursor=42, name="CLONE_SCHEDULER") [2024-04-22 16:23:12.709182] INFO cancel_task (ob_timer.cpp:276) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=71] cancel task(this=0x7f3cfa7fdd20, thread_id=10807, wakeup_time_=1444018670, tasks_num_=7, task=tasktype:N9oceanbase10rootserver34ObMViewRefreshStatsMaintenanceTaskE) [2024-04-22 16:23:12.709226] INFO [CLOG] switch_to_follower_forcedly (ob_role_change_handler.cpp:86) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=14] leader to follower forcedly, current role change handler is(cursor=43, name="MVIEW_MAINTENANCE_SERVICE_LOG_BASE_TYPE") [2024-04-22 16:23:12.709275] WDIAG [PALF] next (log_iterator_impl.h:703) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4008] there is no readable log, set next_min_scn to OB_INVALID_TIMESTAMP(ret=-4008, this={buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358576}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, info={reason:2, log_scn:{val:18446744073709551615, v:3}}, replayable_point_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:12.709351] WDIAG [CLOG] reset_iterator (ob_replay_status.cpp:304) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=73][errcode=-4008] iterator next failed(type_=1, begin_lsn={lsn:358576}, ret=-4008) [2024-04-22 16:23:12.709408] INFO [CLOG] switch_to_follower (ob_replay_status.cpp:850) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=20] replay status switch_to_follower(this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:3, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774192709386, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:358576}, next_to_submit_scn_:{val:1713774177880657608, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358576}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}, begin_lsn={lsn:358576}) [2024-04-22 16:23:12.709490] WDIAG [PALF] next (log_iterator_impl.h:703) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4008] there is no readable log, set next_min_scn to OB_INVALID_TIMESTAMP(ret=-4008, this={buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358576}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, info={reason:2, log_scn:{val:18446744073709551615, v:3}}, replayable_point_scn={val:0, v:0}) [2024-04-22 16:23:12.709574] INFO [CLOG] next_log (ob_replay_status.cpp:259) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=80] next_min_scn is invalid(type_=1, replayable_point={val:0, v:0}, next_min_scn={val:18446744073709551615, v:3}, next_to_submit_scn_={val:1713774177880657608, v:0}, ret=0, iterator_={iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358576}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}) [2024-04-22 16:23:12.709557] INFO [CLOG] switch_leader_to_follower_forcedly_ (ob_role_change_service.cpp:664) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=146] switch_leader_to_follower_forcedly_ success(ret=0, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:12.709706] INFO [CLOG] handle_role_change_event_ (ob_role_change_service.cpp:350) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=146] end log handler role change(ret=0, curr_access_mode=1, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}, retry_ctx={reason:0}) [2024-04-22 16:23:12.709870] INFO [CLOG] handle_role_change_event_ (ob_role_change_service.cpp:351) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=162] begin restore handler role change(curr_access_mode=1, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:12.710006] INFO [CLOG] handle_role_change_event_ (ob_role_change_service.cpp:356) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=130] end restore handler role change(ret=0, curr_access_mode=1, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}, ls={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774197210958129, latest_log_ts:{val:1713774187210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:12.710142] INFO [STORAGE] report_replica_info (ob_ls.cpp:1419) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=118] submit log stream update task(ls_meta_.tenant_id_=1, ls_meta_.ls_id_={id:1}) [2024-04-22 16:23:12.710160] INFO [CLOG] handle (ob_role_change_service.cpp:223) [10818][T1_RCSrv0][T1][Y0-0000000000000000-0-0] [lt=17] end handle_role_change_event_(sequence:=1713774192698691, event={event_type:2, ls_id:{id:1}, dst_addr:"0.0.0.0:0"}) [2024-04-22 16:23:12.728282] INFO keepalive_check (time_wheel.c:76) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=32] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3d01ba4048,dest=-2091378772:2882), old=0, new=3000000 [2024-04-22 16:23:12.728410] INFO keepalive_check (time_wheel.c:76) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=88] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd24d89b8,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:12.739787] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] sock regist: 0x7f3cfd8726b0 fd=216 [2024-04-22 16:23:12.739863] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=59] [ussl] accept new connection, fd:216, src_addr:172.23.88.130:55326 [2024-04-22 16:23:12.739985] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] auth mothod is NONE, the fd will be dispatched, fd:216, src_addr:172.23.88.130:55326 [2024-04-22 16:23:12.740063] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=65] PNIO dispatch fd to certain group, fd:216, gid:0x100000002 [2024-04-22 16:23:12.740194] INFO pkts_sk_init (pkts_sk_factory.h:23) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO set pkts_sk_t sock_id s=0x7f3cd33f6048, s->id=262142 [2024-04-22 16:23:12.740247] INFO pkts_sk_new (pkts_sk_factory.h:51) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=51] PNIO sk_new: s=0x7f3cd33f6048 [2024-04-22 16:23:12.740330] INFO eloop_regist (eloop.c:47) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO sock regist: 0x7f3cd33f6048 fd=216 [2024-04-22 16:23:12.740405] INFO on_accept (listenfd.c:39) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO accept new connection, ns=0x7f3cd33f6048, fd=fd:216:local:172.23.88.129:2882:remote:172.23.88.130:55326 [2024-04-22 16:23:12.740632] WDIAG listenfd_handle_event (listenfd.c:71) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.740936] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C357C-0-0] [lt=149][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:12.740999] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C357C-0-0] [lt=57][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:12.741060] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C357C-0-0] [lt=58][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:12.741136] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C357C-0-0] [lt=72][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:12.754172] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] sock regist: 0x7f3cfd8726b0 fd=217 [2024-04-22 16:23:12.754242] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=56] [ussl] accept new connection, fd:217, src_addr:172.23.88.131:36234 [2024-04-22 16:23:12.754327] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] auth mothod is NONE, the fd will be dispatched, fd:217, src_addr:172.23.88.131:36234 [2024-04-22 16:23:12.754350] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:217, gid:0x100000002 [2024-04-22 16:23:12.754534] INFO pkts_sk_init (pkts_sk_factory.h:23) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=48] PNIO set pkts_sk_t sock_id s=0x7f3cd33f69b8, s->id=655359 [2024-04-22 16:23:12.754611] INFO pkts_sk_new (pkts_sk_factory.h:51) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=75] PNIO sk_new: s=0x7f3cd33f69b8 [2024-04-22 16:23:12.754689] INFO eloop_regist (eloop.c:47) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO sock regist: 0x7f3cd33f69b8 fd=217 [2024-04-22 16:23:12.754741] INFO on_accept (listenfd.c:39) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO accept new connection, ns=0x7f3cd33f69b8, fd=fd:217:local:172.23.88.129:2882:remote:172.23.88.131:36234 [2024-04-22 16:23:12.754871] WDIAG listenfd_handle_event (listenfd.c:71) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.795889] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=102] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:12.798324] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54] get wrs ts(ls_id={id:1}, delta=5376646, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.798404] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:12.798632] INFO [PALF] set_leader_and_epoch_ (log_state_mgr.cpp:1000) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=29] set_leader_and_epoch_(palf_id=1, self="172.23.88.129:2882", leader_="172.23.88.130:2882", leader_epoch_=11, new_leader="172.23.88.130:2882") [2024-04-22 16:23:12.798698] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=63] switch_state(ret=0, palf_id=1, old_role=2, role=2, old_state=2, state=2, old_leader="0.0.0.0:0", leader="172.23.88.130:2882", old_leader_epoch=0, leader_epoch=11, need_next_loop=true, diff=70) [2024-04-22 16:23:12.798743] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=40] switch_state(ret=0, palf_id=1, old_role=2, role=2, old_state=2, state=2, old_leader="172.23.88.130:2882", leader="172.23.88.130:2882", old_leader_epoch=11, leader_epoch=11, need_next_loop=false, diff=2) [2024-04-22 16:23:12.802985] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=71] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:12.818407] INFO keepalive_check (time_wheel.c:76) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=52] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd33f8048,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:12.898382] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:12.898503] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=116][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774192898367) [2024-04-22 16:23:12.898557] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774192698384, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:12.898649] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] get wrs ts(ls_id={id:1}, delta=5476313, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.898706] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:12.924527] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x7f3cfd8726b0 fd=218 [2024-04-22 16:23:12.924598] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=57] [ussl] accept new connection, fd:218, src_addr:172.23.88.131:36236 [2024-04-22 16:23:12.924693] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] auth mothod is NONE, the fd will be dispatched, fd:218, src_addr:172.23.88.131:36236 [2024-04-22 16:23:12.924746] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=41] PNIO dispatch fd to certain group, fd:218, gid:0x100000003 [2024-04-22 16:23:12.924955] INFO pkts_sk_init (pkts_sk_factory.h:23) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=53] PNIO set pkts_sk_t sock_id s=0x7f3d01be69b8, s->id=524287 [2024-04-22 16:23:12.924985] INFO pkts_sk_new (pkts_sk_factory.h:51) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO sk_new: s=0x7f3d01be69b8 [2024-04-22 16:23:12.925028] INFO eloop_regist (eloop.c:47) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO sock regist: 0x7f3d01be69b8 fd=218 [2024-04-22 16:23:12.925047] INFO on_accept (listenfd.c:39) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO accept new connection, ns=0x7f3d01be69b8, fd=fd:218:local:172.23.88.129:2882:remote:172.23.88.131:36236 [2024-04-22 16:23:12.925108] WDIAG listenfd_handle_event (listenfd.c:71) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:12.996628] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=93] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:12.998524] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72] get wrs ts(ls_id={id:1}, delta=5576439, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:12.998598] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.003221] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=76] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:13.034384] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] sock regist: 0x7f3cfd8726b0 fd=219 [2024-04-22 16:23:13.034514] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=114] [ussl] accept new connection, fd:219, src_addr:172.23.88.131:36238 [2024-04-22 16:23:13.034605] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] auth mothod is NONE, the fd will be dispatched, fd:219, src_addr:172.23.88.131:36238 [2024-04-22 16:23:13.034630] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] PNIO dispatch fd to certain group, fd:219, gid:0x100000005 [2024-04-22 16:23:13.034718] INFO pkts_sk_init (pkts_sk_factory.h:23) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=16] PNIO set pkts_sk_t sock_id s=0x7f3d01be7328, s->id=589821 [2024-04-22 16:23:13.034781] INFO pkts_sk_new (pkts_sk_factory.h:51) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=62] PNIO sk_new: s=0x7f3d01be7328 [2024-04-22 16:23:13.034862] INFO eloop_regist (eloop.c:47) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO sock regist: 0x7f3d01be7328 fd=219 [2024-04-22 16:23:13.034917] INFO on_accept (listenfd.c:39) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=31] PNIO accept new connection, ns=0x7f3d01be7328, fd=fd:219:local:172.23.88.129:2882:remote:172.23.88.131:36238 [2024-04-22 16:23:13.035042] WDIAG listenfd_handle_event (listenfd.c:71) [10463][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:13.049779] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x7f3cfd8726b0 fd=220 [2024-04-22 16:23:13.049856] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=59] [ussl] accept new connection, fd:220, src_addr:172.23.88.131:36240 [2024-04-22 16:23:13.049975] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=28] [ussl] auth mothod is NONE, the fd will be dispatched, fd:220, src_addr:172.23.88.131:36240 [2024-04-22 16:23:13.050036] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=46] PNIO dispatch fd to certain group, fd:220, gid:0x100000001 [2024-04-22 16:23:13.050201] INFO pkts_sk_init (pkts_sk_factory.h:23) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO set pkts_sk_t sock_id s=0x7f3d01a04708, s->id=589822 [2024-04-22 16:23:13.050271] INFO pkts_sk_new (pkts_sk_factory.h:51) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=67] PNIO sk_new: s=0x7f3d01a04708 [2024-04-22 16:23:13.050363] INFO eloop_regist (eloop.c:47) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=26] PNIO sock regist: 0x7f3d01a04708 fd=220 [2024-04-22 16:23:13.050417] INFO on_accept (listenfd.c:39) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=31] PNIO accept new connection, ns=0x7f3d01a04708, fd=fd:220:local:172.23.88.129:2882:remote:172.23.88.131:36240 [2024-04-22 16:23:13.050578] WDIAG listenfd_handle_event (listenfd.c:71) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:13.098625] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=77][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:13.098695] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774193098612) [2024-04-22 16:23:13.098720] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774192898590, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:13.098777] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=5676677, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.098815] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.124656] INFO keepalive_check (time_wheel.c:76) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=32] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd1aad358,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:13.124782] INFO keepalive_check (time_wheel.c:76) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=71] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd33d4048,dest=-2091378772:2882), old=0, new=3000000 [2024-04-22 16:23:13.126192] INFO keepalive_check (time_wheel.c:76) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=33] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3d01ba49e8,dest=-2091378772:2882), old=0, new=3000000 [2024-04-22 16:23:13.126287] INFO keepalive_check (time_wheel.c:76) [10459][pnio1][T0][Y0-0000000000000000-0-0] [lt=55] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3d01bfa048,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:23:13.190198] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.190278] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.190545] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.190598] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.190613] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=43] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.190667] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.191117] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:222) [10510][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=59] [ASYNC TASK QUEUE](queue_size=0) [2024-04-22 16:23:13.191128] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.191504] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.191536] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.191721] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=63] gc stale ls task succ [2024-04-22 16:23:13.191814] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10427, mem_used=79030272) [2024-04-22 16:23:13.191866] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.191957] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=58] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:13.192074] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [10543][Blacklist][T0][Y0-0000000000000000-0-0] [lt=175] blacklist_loop exec finished(cost_time=62, is_enabled=true, send_cnt=0) [2024-04-22 16:23:13.192231] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=48] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.192554] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=50] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.192668] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=281] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:13.192826] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.192935] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.193357] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=50] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.193789] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=62] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.193930] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] [COMPACTION DAG STATUS] (start_time=1713774177718863, end_time=1713774193193927, dag_status={}) [2024-04-22 16:23:13.193981] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=47] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.194005] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.194020] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.193998] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=69] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.194034] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.194048] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.194061] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.194070] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=52] PNIO [ratelimit] time: 1713774193194068, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001183, add_bytes: 0 [2024-04-22 16:23:13.194075] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.194102] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=26] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.194116] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:13.194130] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194167] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=36] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194184] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194200] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194216] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194232] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194291] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194315] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194331] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194364] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:0, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194380] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194396] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194400] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:13.194412] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194454] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:13.194428] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194484] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:13.194483] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=53] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194500] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:13.194501] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194517] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194534] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194534] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2024-04-22 16:23:13.194549] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194599] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=48] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194658] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194677] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194694] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194710] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194726] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194742] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194758] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194773] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194789] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194818] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194835] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194851] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194867] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194883] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194899] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194914] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194930] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194956] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194972] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.194988] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195004] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195020] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195036] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195052] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195068] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195106] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=36] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195132] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195157] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=24] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195181] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195216] WDIAG [STORAGE.TRANS] wait (ob_ts_mgr.cpp:208) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4012] ObTsSyncGetTsCbTask cond wait failed(ret=-4012) [2024-04-22 16:23:13.195244] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=62] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195261] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195257] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:940) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33][errcode=-4012] failed to wait ObTsSyncGetTsCbTask(ret=-4012, tenant_id=1, timeout_us=1000000) [2024-04-22 16:23:13.195277] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195281] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1808) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4012] get gts fail(ret=-4012, expire_ts=1713774214757141, now=[mts=1713774192194959]) [2024-04-22 16:23:13.195294] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195299] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1822) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1713774214757141, now=[mts=1713774192194959], now0=[mts=1713774192194959], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-04-22 16:23:13.195320] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195337] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195354] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195370] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:13.195389] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3583) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2024-04-22 16:23:13.195402] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2024-04-22 16:23:13.195327] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:657) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185670274, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.195415] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2024-04-22 16:23:13.195471] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=54] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2024-04-22 16:23:13.195482] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2024-04-22 16:23:13.195492] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2024-04-22 16:23:13.195494] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.195430] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:736) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=96][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}) [2024-04-22 16:23:13.195503] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2024-04-22 16:23:13.195525] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2024-04-22 16:23:13.195535] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2024-04-22 16:23:13.195545] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2024-04-22 16:23:13.195558] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4271) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(dag_cnt=0, total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=0) [2024-04-22 16:23:13.195525] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:566) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=92][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185670274, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.195585] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:636) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=57] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f3cd9453038, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, savepoint=0, tx_desc={this:0x7f3ccb8f1150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774185670274, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1713774184827148, use_das=false, nested_level=0, session={this:0x7f3cd22061a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb8f1150}, plan=0x7f3ce5408050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-04-22 16:23:13.195717] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=129][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[cnt:1, {table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-04-22 16:23:13.195744] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4283] fail start stmt(ret=-4283) [2024-04-22 16:23:13.195757] WDIAG [SQL] open (ob_result_set.cpp:158) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4283] execute plan failed(ret=-4283) [2024-04-22 16:23:13.195771] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4283] open result set failed(ret=-4283) [2024-04-22 16:23:13.195829] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=54][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.195849] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=41, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.195895] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=41000, remain_us=21631247, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4283, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.195964] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=108][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:525, tid:10410}]) [2024-04-22 16:23:13.195969] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.196616] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.196693] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=0] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:13.197040] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=72] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:13.197625] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=50] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.197705] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED3B-0-0] [lt=4][errcode=-4006] not init(ret=-4006) [2024-04-22 16:23:13.197694] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.197852] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=5][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:13.197873] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.197844] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:466) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED3B-0-0] [lt=19] finish scan map holding items(scan_cnt=0) [2024-04-22 16:23:13.197928] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:13.197944] INFO [CLOG] handle_sync_request (log_request_handler.cpp:513) [11430][T1_L0_G1][T1][YB42AC175881-000616AB0B4FEC10-0-0] [lt=74] i am not leader(ret=-4038, palf_id=1, req={get_type:1}, role=2, is_pending_state=false) [2024-04-22 16:23:13.197961] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:13.197993] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=29][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:13.198020] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=24][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:13.198042] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:13.198038] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED3C-0-0] [lt=77] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:13.198055] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:13.198072] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:13.198085] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:13.198096] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:13.198107] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:13.198118] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:13.198130] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:13.198091] INFO [SHARE] run2 (ob_async_task_queue.cpp:169) [10498][RSInspectTask][T0][Y0-0000000000000000-0-0] [lt=177] async task queue stop [2024-04-22 16:23:13.198142] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:13.198164] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=20][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:13.198190] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:13.198204] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.198218] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.198230] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:13.198241] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:13.198257] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:13.198268] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.198249] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4038] execute rpc fail(addr="172.23.88.129:2882", pcode=5407, ret=-4038, timeout=2000000) [2024-04-22 16:23:13.198294] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:13.198300] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4038] [2024-04-22 16:23:13.198320] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.198332] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.198334] WDIAG [CLOG] post_sync_request_to_server_ (log_net_service.h:347) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] ObLogRpc post_sync_request failed(ret=-4038, palf_id=1, req={get_type:1}, server="172.23.88.129:2882") [2024-04-22 16:23:13.198364] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5195) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4038] get_palf_max_scn failed(ret=-4038, palf_id=1) [2024-04-22 16:23:13.198342] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:13.198407] INFO run1 (ob_timer.cpp:479) [11481][GlobalCtxTimer][T0][Y0-0000000000000000-0-0] [lt=29] timer thread exit(this=0x7f3d1bcb39c0, thread_id=11481) [2024-04-22 16:23:13.198427] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED80-0-0] [lt=57][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:13.198469] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5155) [10720][T1_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] get_palf_max_scn failed(ret=-4038, self="172.23.88.129:2882", palf_id=1) [2024-04-22 16:23:13.198483] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=51][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:13.198422] WDIAG [STORAGE.TRANS] wait (ob_ts_mgr.cpp:208) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=4][errcode=-4012] ObTsSyncGetTsCbTask cond wait failed(ret=-4012) [2024-04-22 16:23:13.198499] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:13.198509] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:940) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=80][errcode=-4012] failed to wait ObTsSyncGetTsCbTask(ret=-4012, tenant_id=1, timeout_us=1000000) [2024-04-22 16:23:13.198514] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:13.198525] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1808) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=15][errcode=-4012] get gts fail(ret=-4012, expire_ts=1713774185671139, now=[mts=1713774192198289]) [2024-04-22 16:23:13.198530] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=14][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:13.198534] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:13.198537] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1822) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1713774185671139, now=[mts=1713774192198289], now0=[mts=1713774192198289], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-04-22 16:23:13.198550] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=18][errcode=-5019] query failed(ret=-5019, conn=0x7f3cbe430050, start=1713774193197494, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:13.198577] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=25][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:13.198556] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:657) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=16][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188245791, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.198595] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=14][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:13.198613] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:736) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=52][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}) [2024-04-22 16:23:13.198655] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:566) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=41][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188245791, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.198713] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=18][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:13.198741] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED80-0-0] [lt=26][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:13.198718] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:636) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=61] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f3cff1ce058, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, savepoint=0, tx_desc={this:0x7f3cd2383310, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188245791, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1713774183741717, use_das=false, nested_level=0, session={this:0x7f3cf3bc01a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cd2383310}, plan=0x7f3ce540e050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-04-22 16:23:13.198786] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:13.198830] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=109][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[cnt:1, {table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-04-22 16:23:13.198868] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=35][errcode=-4283] fail start stmt(ret=-4283) [2024-04-22 16:23:13.198883] WDIAG [SQL] open (ob_result_set.cpp:158) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=13][errcode=-4283] execute plan failed(ret=-4283) [2024-04-22 16:23:13.198895] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=10][errcode=-4283] open result set failed(ret=-4283) [2024-04-22 16:23:13.198874] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774193198769) [2024-04-22 16:23:13.198909] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=12][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:13.198926] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=15][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=62, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.198922] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774193098736, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:13.198931] WDIAG [STORAGE.TRANS] wait (ob_ts_mgr.cpp:208) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=8][errcode=-4012] ObTsSyncGetTsCbTask cond wait failed(ret=-4012) [2024-04-22 16:23:13.198941] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=13][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:62, local_retry_times:62, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:23:13.198961] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:940) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=23][errcode=-4012] failed to wait ObTsSyncGetTsCbTask(ret=-4012, tenant_id=1, timeout_us=1000000) [2024-04-22 16:23:13.198980] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1808) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=18][errcode=-4012] get gts fail(ret=-4012, expire_ts=1713774194125630, now=[mts=1713774192198789]) [2024-04-22 16:23:13.198992] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=5776776, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.198995] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1822) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=14][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1713774194125630, now=[mts=1713774192198789], now0=[mts=1713774192198789], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-04-22 16:23:13.199024] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=26][errcode=-4283] result set close failed(ret=-4283) [2024-04-22 16:23:13.199043] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=17][errcode=-4283] result set close failed(ret=-4283) [2024-04-22 16:23:13.199020] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.199055] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=11][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-04-22 16:23:13.199092] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=27][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:23:13.199085] INFO [SQL.ENG] check_dummy_location_credible (ob_px_tenant_target_monitor.cpp:181) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=18] dummy location not credible, need refresh [2024-04-22 16:23:13.199018] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:657) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=19][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f3ccb931150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240321, assoc_session_id:3221240321, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188245791, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.199107] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=11] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=9457389) [2024-04-22 16:23:13.199110] INFO [SQL.ENG] refresh_dummy_location (ob_px_tenant_target_monitor.cpp:232) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=23] waiting for refresh locatition cache for target_monitor(tenant_id_=1, refresh_ctrl=14) [2024-04-22 16:23:13.199138] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=30][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:23:13.199141] WDIAG [SQL.ENG] get_dummy_leader (ob_px_tenant_target_monitor.cpp:165) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] refresh dummy location cache(ret=0) [2024-04-22 16:23:13.199130] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:736) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=106][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f3ccc89e1f8, id:3221240321, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb931150}) [2024-04-22 16:23:13.199156] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=14][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:13.199173] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=16][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:23:13.199188] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=14][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:13.199219] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=28][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:13.199163] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:566) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=32][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221240321, *tx_desc={this:0x7f3ccb931150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240321, assoc_session_id:3221240321, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188245791, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.199247] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=22][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:13.199303] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-04-22 16:23:13.199253] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:636) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=87] start stmt(ret=-4283, auto_commit=true, session_id=3221240321, snapshot={this:0x7f3cd2009368, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, savepoint=0, tx_desc={this:0x7f3ccb931150, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240321, assoc_session_id:3221240321, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188245791, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1713774192197489, use_das=false, nested_level=0, session={this:0x7f3ccc89e1f8, id:3221240321, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3ccb931150}, plan=0x7f3ce5474050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-04-22 16:23:13.199355] WDIAG [SQL.EXE] auto_start_phy_trans (ob_remote_executor_processor.cpp:236) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=99][errcode=-4283] fail to start stmt(ret=-4283) [2024-04-22 16:23:13.199369] WDIAG [SQL.EXE] execute_remote_plan (ob_remote_executor_processor.cpp:450) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=13][errcode=-4283] fail to begin trans(ret=-4283) [2024-04-22 16:23:13.199343] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [10569][TsMgr][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="172.23.88.129:2882", msg={tenant_id:1, srr:[mts=1713774193199274], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:13.199431] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=21][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:23:13.199406] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="172.23.88.129:2882", msg={tenant_id:1, srr:[mts=1713774193199274], range_size:1, sender:"172.23.88.129:2882"}) [2024-04-22 16:23:13.199481] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=48][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:13.199493] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=11][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:23:13.199478] WDIAG [STORAGE.TRANS] wait (ob_ts_mgr.cpp:208) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=6][errcode=-4012] ObTsSyncGetTsCbTask cond wait failed(ret=-4012) [2024-04-22 16:23:13.199514] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:940) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=27][errcode=-4012] failed to wait ObTsSyncGetTsCbTask(ret=-4012, tenant_id=1, timeout_us=1000000) [2024-04-22 16:23:13.199546] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=9][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:13.199542] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1808) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=26][errcode=-4012] get gts fail(ret=-4012, expire_ts=1713774222128419, now=[mts=1713774192199300]) [2024-04-22 16:23:13.199565] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=17][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:13.199566] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1822) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=22][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1713774222128419, now=[mts=1713774192199300], now0=[mts=1713774192199300], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-04-22 16:23:13.199386] WDIAG [SQL.EXE] execute_with_sql (ob_remote_executor_processor.cpp:690) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=14][errcode=-4283] execute remote plan failed(ret=-4283, task={tenant_schema_version:1, sys_schema_version:1, runner_svr:"172.23.88.129:2882", ctrl_svr:"172.23.88.131:2882", task_id:{ob_job_id:{ob_execution_id:{server:"172.23.88.131:2882", execution_id:18446744073709551615, task_type:0, hash:4219408603045787357}, job_id:18446744073709551615}, task_id:0, task_cnt:0}, remote_sql_info:{use_ps:true, is_batched_stmt:false, is_original_ps_mode:false, ps_param_cnt:1, remote_sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name", ps_params:[{obj:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci", coercibility:"COERCIBLE"}, accuracy:{length:17, precision:1, scale:-1}, flag:1, raw_text_pos:-1, raw_text_len:-1, param_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"COERCIBLE"}}]}, snapshot:{this:0x7f3cd2004200, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}}, exec_ctx_.get_das_ctx().get_snapshot()={this:0x7f3cd2009368, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}) [2024-04-22 16:23:13.199566] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=75] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1713774185662216], gts:1713774187494514341, latest_srr:[mts=1713774193199274]}) [2024-04-22 16:23:13.199579] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:521) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=13][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-04-22 16:23:13.199600] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:657) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=30][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f3cc5e215b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240322, assoc_session_id:3221240322, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188246886, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.199755] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:736) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=147][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f3cd1168ce8, id:3221240322, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cc5e215b0}) [2024-04-22 16:23:13.199723] TRACE [TRACE] base_after_process (ob_remote_executor_processor.cpp:816) [10949][T1_L0_G0][T1][YB42AC175883-000616AB09773260-0-0] [lt=214] [slow remote task](TRACE=begin_ts=1713774192197767 2024-04-22 08:23:12.197767|[process_begin] u=0 in_queue_time:83, receive_ts:1713774192197683, enqueue_ts:1713774192197685|[start_rpc] u=0 addr:{ip:"172.23.88.131", port:36178}, pcode:1308|[query_begin] u=1 trace_id:YB42AC175883-000616AB09773260-0-0|[before_processor_run] u=16 |[exec_remote_plan_begin] u=882 |[calc_partition_location_begin] u=86 |[tl_calc_part_id_end] u=1 |[get_location_cache_begin] u=0 |[get_location_cache_end] u=2 |[calc_partition_location_end] u=0 |[pc_choose_plan] u=2 |[plan_id] u=2 plan_id:101|[execute_task] u=2 task:{task_id:{job_id:{execution_id:{addr:{ip:"172.23.88.131", port:2882}, execution_id:18446744073709551615}, job_id:18446744073709551615}, task_id:0}, runner_svr:{ip:"172.23.88.129", port:2882}, ctrl_svr:{ip:"172.23.88.131", port:2882}}, stmt_type:1|[execute_task] u=1 task:{task_id:{job_id:{execution_id:{addr:{ip:"172.23.88.131", port:2882}, execution_id:18446744073709551615}, job_id:18446744073709551615}, task_id:0}, runner_svr:{ip:"172.23.88.129", port:2882}, ctrl_svr:{ip:"172.23.88.131", port:2882}}, stmt_type:1|[get_location_cache_begin] u=1000842 |[exec_remote_plan_end] u=32 |total_timeu=1001869) [2024-04-22 16:23:13.199804] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:566) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=47][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221240322, *tx_desc={this:0x7f3cc5e215b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240322, assoc_session_id:3221240322, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188246886, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.199888] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=112] PNIO [ratelimit] time: 1713774193199884, bytes: 66111271, bw: 0.013838 MB/s, add_ts: 1000241, add_bytes: 14514 [2024-04-22 16:23:13.199894] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:636) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=88] start stmt(ret=-4283, auto_commit=true, session_id=3221240322, snapshot={this:0x7f3cc5409368, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, savepoint=0, tx_desc={this:0x7f3cc5e215b0, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240322, assoc_session_id:3221240322, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774188246886, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1713774192198426, use_das=false, nested_level=0, session={this:0x7f3cd1168ce8, id:3221240322, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f3cc5e215b0}, plan=0x7f3ce54aa050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-04-22 16:23:13.200056] WDIAG [SQL.EXE] auto_start_phy_trans (ob_remote_executor_processor.cpp:236) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=158][errcode=-4283] fail to start stmt(ret=-4283) [2024-04-22 16:23:13.200077] WDIAG [SQL.EXE] execute_remote_plan (ob_remote_executor_processor.cpp:450) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=20][errcode=-4283] fail to begin trans(ret=-4283) [2024-04-22 16:23:13.200090] WDIAG [SQL.EXE] execute_with_sql (ob_remote_executor_processor.cpp:690) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=10][errcode=-4283] execute remote plan failed(ret=-4283, task={tenant_schema_version:1, sys_schema_version:1, runner_svr:"172.23.88.129:2882", ctrl_svr:"172.23.88.130:2882", task_id:{ob_job_id:{ob_execution_id:{server:"172.23.88.130:2882", execution_id:18446744073709551615, task_type:0, hash:4219408603045787357}, job_id:18446744073709551615}, task_id:0, task_cnt:0}, remote_sql_info:{use_ps:true, is_batched_stmt:false, is_original_ps_mode:false, ps_param_cnt:2, remote_sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = ? AND COLUMN_NAME = ?", ps_params:[{obj:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci", coercibility:"COERCIBLE"}, accuracy:{length:17, precision:1, scale:-1}, flag:1, raw_text_pos:-1, raw_text_len:-1, param_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"COERCIBLE"}}, {obj:{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci", coercibility:"COERCIBLE"}, accuracy:{length:15, precision:1, scale:-1}, flag:1, raw_text_pos:-1, raw_text_len:-1, param_meta:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"COERCIBLE"}}]}, snapshot:{this:0x7f3cc5404200, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}}, exec_ctx_.get_das_ctx().get_snapshot()={this:0x7f3cc5409368, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}) [2024-04-22 16:23:13.200283] WDIAG [STORAGE.TRANS] wait (ob_ts_mgr.cpp:208) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=6][errcode=-4012] ObTsSyncGetTsCbTask cond wait failed(ret=-4012) [2024-04-22 16:23:13.200317] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:940) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=28][errcode=-4012] failed to wait ObTsSyncGetTsCbTask(ret=-4012, tenant_id=1, timeout_us=1000000) [2024-04-22 16:23:13.200337] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1808) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=18][errcode=-4012] get gts fail(ret=-4012, expire_ts=1713774221273715, now=[mts=1713774192200137]) [2024-04-22 16:23:13.200363] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1822) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=25][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1713774221273715, now=[mts=1713774192200137], now0=[mts=1713774192200137], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-04-22 16:23:13.200253] TRACE [TRACE] base_after_process (ob_remote_executor_processor.cpp:816) [10891][T1_L0_G0][T1][YB42AC175882-000616AB07FC2F07-0-0] [lt=97] [slow remote task](TRACE=begin_ts=1713774192198672 2024-04-22 08:23:12.198672|[process_begin] u=0 in_queue_time:75, receive_ts:1713774192198597, enqueue_ts:1713774192198664|[start_rpc] u=1 addr:{ip:"172.23.88.130", port:55290}, pcode:1308|[query_begin] u=1 trace_id:YB42AC175882-000616AB07FC2F07-0-0|[before_processor_run] u=7 |[exec_remote_plan_begin] u=513 |[calc_partition_location_begin] u=77 |[tl_calc_part_id_end] u=1 |[get_location_cache_begin] u=0 |[get_location_cache_end] u=2 |[calc_partition_location_end] u=0 |[pc_choose_plan] u=2 |[plan_id] u=3 plan_id:102|[execute_task] u=1 task:{task_id:{job_id:{execution_id:{addr:{ip:"172.23.88.130", port:2882}, execution_id:18446744073709551615}, job_id:18446744073709551615}, task_id:0}, runner_svr:{ip:"172.23.88.129", port:2882}, ctrl_svr:{ip:"172.23.88.130", port:2882}}, stmt_type:1|[execute_task] u=1 task:{task_id:{job_id:{execution_id:{addr:{ip:"172.23.88.130", port:2882}, execution_id:18446744073709551615}, job_id:18446744073709551615}, task_id:0}, runner_svr:{ip:"172.23.88.129", port:2882}, ctrl_svr:{ip:"172.23.88.130", port:2882}}, stmt_type:1|[get_location_cache_begin] u=1000908 |[exec_remote_plan_end] u=12 |total_timeu=1001529) [2024-04-22 16:23:13.200378] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:657) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=13][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f3cd155c970, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240323, assoc_session_id:3221240323, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774192196433, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.200418] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:736) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=37][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x7f3cd115fe98, id:3221240323, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155c970}) [2024-04-22 16:23:13.200457] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:566) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=38][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221240323, *tx_desc={this:0x7f3cd155c970, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240323, assoc_session_id:3221240323, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774192196433, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-04-22 16:23:13.200490] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:636) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=31] start stmt(ret=-4283, auto_commit=true, session_id=3221240323, snapshot={this:0x7f3cbde09a28, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[], committed:false}, savepoint=0, tx_desc={this:0x7f3cd155c970, tx_id:{txid:0}, state:1, addr:"172.23.88.129:2882", tenant_id:1, session_id:3221240323, assoc_session_id:3221240323, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1713774185670275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:17180065793, seq_base:1713774192196433, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=2, stmt_type=1, has_for_update=false, query_start_time=1713774191343735, use_das=false, nested_level=0, session={this:0x7f3cd115fe98, id:3221240323, deser:true, tenant:"", tenant_id:1, effective_tenant:"", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f3cd155c970}, plan=0x7f3cbde0a3a8, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-04-22 16:23:13.200555] WDIAG [SQL.EXE] auto_start_phy_trans (ob_remote_executor_processor.cpp:236) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=63][errcode=-4283] fail to start stmt(ret=-4283) [2024-04-22 16:23:13.200563] WDIAG [SQL.EXE] execute_remote_plan (ob_remote_executor_processor.cpp:450) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=7][errcode=-4283] fail to begin trans(ret=-4283) [2024-04-22 16:23:13.200571] WDIAG [SQL.EXE] process (ob_remote_executor_processor.cpp:990) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=6][errcode=-4283] execute remote plan failed(ret=-4283, local_tenant_schema_version=1, exec_ctx_.get_task_exec_ctx().get_query_tenant_begin_schema_version()=1) [2024-04-22 16:23:13.200626] TRACE [TRACE] base_after_process (ob_remote_executor_processor.cpp:816) [10882][T1_L5_G0][T1][YB42AC175883-000616AB08373247-0-0] [lt=8] [slow remote task](TRACE=begin_ts=1713774192199232 2024-04-22 08:23:12.199232|[process_begin] u=0 in_queue_time:23, receive_ts:1713774192199208, enqueue_ts:1713774192199209|[start_rpc] u=0 addr:{ip:"172.23.88.131", port:36176}, pcode:1282|[query_begin] u=1 trace_id:YB42AC175883-000616AB08373247-0-0|[before_processor_run] u=10 |[exec_remote_plan_begin] u=884 |[execute_task] u=1 task:{task_id:{job_id:{execution_id:{addr:{ip:"172.23.88.131", port:2882}, execution_id:18446744073709551615}, job_id:36161}, task_id:0}, runner_svr:{ip:"172.23.88.129", port:2882}, ctrl_svr:{ip:"172.23.88.131", port:2882}}, stmt_type:1|[execute_task] u=1 task:{task_id:{job_id:{execution_id:{addr:{ip:"172.23.88.131", port:2882}, execution_id:18446744073709551615}, job_id:36161}, task_id:0}, runner_svr:{ip:"172.23.88.129", port:2882}, ctrl_svr:{ip:"172.23.88.131", port:2882}}, stmt_type:1|[process_ret] u=1000451 process_ret:-4283|[exec_remote_plan_end] u=10 |total_timeu=1001358) [2024-04-22 16:23:13.200894] INFO [SHARE.LOCATION] merge_leader_from (ob_location_struct.cpp:488) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED6F-0-0] [lt=4] [LS_LOCATION] leader will be changed(new_leader={server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:8}, old_leader={server:"172.23.88.129:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}, exist_replica=NULL, old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774192194330, replica_locations:[{server:"172.23.88.129:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:7}, {server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}) [2024-04-22 16:23:13.201693] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=0] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774191670314, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:23:13.201842] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:582) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=109] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2024-04-22 16:23:13.201869] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:221) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=22] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=307) [2024-04-22 16:23:13.202133] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:13.202210] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_rootservice_event_history, table_name.ptr()="data_size:31, data:5F5F616C6C5F726F6F74736572766963655F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:13.202292] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=77][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_rootservice_event_history, ret=-5019) [2024-04-22 16:23:13.202316] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=22][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_rootservice_event_history, db_name=oceanbase) [2024-04-22 16:23:13.202334] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_rootservice_event_history) [2024-04-22 16:23:13.202354] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:13.202367] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:13.202383] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_rootservice_event_history' doesn't exist [2024-04-22 16:23:13.202395] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:13.202407] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=11][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:13.202419] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=11][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:13.202430] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:13.202488] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=55][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:13.202511] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:13.202539] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=24][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.202554] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.202566] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:13.202586] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=15][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:13.202607] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=19][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:13.202624] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=15][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.202659] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=30][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:13.202695] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=29][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.202707] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.202716] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:13.202796] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10504][EvtHisUpdTask][T1][YB42AC175881-000616AB09BFEBC0-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:13.202825] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10504][EvtHisUpdTask][T0][YB42AC175881-000616AB09BFEBC0-0-0] [lt=24][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:13.202840] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10504][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:13.202881] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10504][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:13.202906] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10504][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:13.202925] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10504][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:13.202940] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10504][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3d1177a050, start=1713774193199774, sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:13.203022] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=14][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:13.203009] WDIAG [SHARE] process_task (ob_event_history_table_operator.cpp:324) [10504][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute sql failed(sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:13.203052] WDIAG [SHARE] process (ob_event_history_table_operator.cpp:170) [10504][EvtHisUpdTask][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-5019] process_task failed(sql=INSERT INTO __all_rootservice_event_history (gmt_create, module, event, name1, value1, value2, value3, value4, value5, value6, rs_svr_ip, rs_svr_port) VALUES (usec_to_time(1713774193199654), 'root_service', 'finish_wait_stop', 'cost', 990058, '', '', '', '', '', '172.23.88.129', 2882), is_delete=false, ret=-5019, ret="OB_TABLE_NOT_EXIST", create_time_=1713774193199689) [2024-04-22 16:23:13.203067] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=34] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:13.203090] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=20][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:13.203103] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:13.203127] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=23][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:13.203140] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:13.203148] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:13.203158] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:13.203166] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:13.203172] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:13.203179] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:13.203187] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:13.203194] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:13.203201] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:13.203208] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:13.203222] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=7][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:13.203230] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.203245] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.203253] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=7][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:13.203260] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:13.203268] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:13.203276] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.203291] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=12][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:13.203307] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=12][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.203314] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.203320] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:13.203333] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:13.203342] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:13.203356] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:13.203365] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:13.203373] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:13.203380] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] query failed(ret=-5019, conn=0x7f3cc12b0050, start=1713774193202890, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:13.203391] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:13.203399] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:13.203482] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=9][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:13.203494] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=10][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:13.203416] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=63] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:13.203503] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=9][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:13.203512] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=8][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:13.203521] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=6][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:13.203539] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=16][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:13.203547] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7C-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:13.205858] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:236) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEE-0-0] [lt=46][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-04-22 16:23:13.205943] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:599) [10477][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=73][errcode=-4023] fail to batch process task(ret=-4023) [2024-04-22 16:23:13.205965] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:499) [10477][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-04-22 16:23:13.209515] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFCAA-0-0] [lt=26][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774192710127}) [2024-04-22 16:23:13.236205] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] [ussl] sock regist: 0x7f3cfd8723b0 fd=221 [2024-04-22 16:23:13.236283] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=63] [ussl] accept new connection, fd:221, src_addr:172.23.88.130:55328 [2024-04-22 16:23:13.236374] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] auth mothod is NONE, the fd will be dispatched, fd:221, src_addr:172.23.88.130:55328 [2024-04-22 16:23:13.236496] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] PNIO dispatch fd to certain group, fd:221, gid:0x100000004 [2024-04-22 16:23:13.236628] INFO pkts_sk_init (pkts_sk_factory.h:23) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO set pkts_sk_t sock_id s=0x7f3cd33f7328, s->id=655357 [2024-04-22 16:23:13.236686] INFO pkts_sk_new (pkts_sk_factory.h:51) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=55] PNIO sk_new: s=0x7f3cd33f7328 [2024-04-22 16:23:13.236770] INFO eloop_regist (eloop.c:47) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO sock regist: 0x7f3cd33f7328 fd=221 [2024-04-22 16:23:13.236822] INFO on_accept (listenfd.c:39) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO accept new connection, ns=0x7f3cd33f7328, fd=fd:221:local:172.23.88.129:2882:remote:172.23.88.130:55328 [2024-04-22 16:23:13.236946] WDIAG listenfd_handle_event (listenfd.c:71) [10462][pnio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:13.237011] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:41, local_retry_times:41, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-04-22 16:23:13.237087] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C357F-0-0] [lt=329][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:13.237096] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=53][errcode=-4283] result set close failed(ret=-4283) [2024-04-22 16:23:13.237152] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=53][errcode=-4283] result set close failed(ret=-4283) [2024-04-22 16:23:13.237167] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-04-22 16:23:13.237166] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C357F-0-0] [lt=74][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:13.237197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4283, ret=-4283) [2024-04-22 16:23:13.237223] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C357F-0-0] [lt=54][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:13.237294] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2192) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31] need check schema(new_schema_version=1, cached_tenant_schema_version=1, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = ? AND COLUMN_NAME = ?) [2024-04-22 16:23:13.237260] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C357F-0-0] [lt=35][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:13.239831] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [2024-04-22 16:23:13.239866] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.239889] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.239903] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.239915] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.239925] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.239934] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.239981] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.240039] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=51][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.240052] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.240107] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=52][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=42, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.240162] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38] will sleep(sleep_us=42000, remain_us=21586981, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.245682] INFO [PALF] can_receive_config_log (log_config_mgr.cpp:2029) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=42] can_receive_config_log(bool_ret=true, palf_id=1, self="172.23.88.129:2882", leader="172.23.88.130:2882", meta={version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, log_ms_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}) [2024-04-22 16:23:13.246045] INFO [PALF] pre_check_for_config_log (log_sliding_window.cpp:3825) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=355] next log_task is invalid, no need truncate(next_log_id=2942, palf_id=1, self="172.23.88.129:2882", log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}) [2024-04-22 16:23:13.246377] INFO [PALF] update_election_meta_ (log_config_mgr.cpp:1834) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=155] update_election_meta_ success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882", log_ms_meta={version:3, proposal_id:7, prev:{version:1, config:{config_version:{proposal_id:6, config_seq:7}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_mode_pid:7}, new_election_member_list={addr_list:["172.23.88.129:2882", "172.23.88.130:2882", "172.23.88.131:2882"], membership_version:{proposal_id:8, config_seq:9}, replica_num:3}) [2024-04-22 16:23:13.246569] INFO [PALF] serialize (log_meta.cpp:185) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=159] LogMeta serialize(*this={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, buf="", buf=0x7f3cc36d2550, pos=779) [2024-04-22 16:23:13.246712] INFO [PALF] serialize_log_meta_ (log_engine.cpp:1528) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=136] serialize_log_meta_ success(ret=0, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_entry={header:{magic:19533, version:1, data_len:4072, data_checksum:1614797911, header_checksum:3417868676}, buf_:0x7f3cc36d2550}, log_meta_entry_header_len=24) [2024-04-22 16:23:13.246846] INFO [PALF] submit_io_task (log_io_worker.cpp:149) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=119] submit flush meta task success(io_task={LogIOTask:{BaseClass:"LogIOTask", palf_id:1, palf_epoch:1, create_task_ts:1713774193246547, push_cb_into_cb_pool_ts:-1, submit_seq_:31}, flush_meta_cb_ctx:{type:"CHANGE_CONFIG_META", proposal_id:8, config_version:{proposal_id:8, config_seq:9}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}}) [2024-04-22 16:23:13.246943] INFO [PALF] submit_flush_meta_task_ (log_engine.cpp:1287) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=71] submit_flush_meta_task_ success(flush_meta_cb_ctx={type:"CHANGE_CONFIG_META", proposal_id:8, config_version:{proposal_id:8, config_seq:9}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}, log_meta={version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}) [2024-04-22 16:23:13.246999] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=4, stat_interval=1000000, avg (count/cost)=5788, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:13.247071] INFO [PALF] submit_flush_change_config_meta_task (log_engine.cpp:416) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=126] submit_flush_change_config_meta_task success(ret=0, flush_meta_cb_ctx={type:"CHANGE_CONFIG_META", proposal_id:8, config_version:{proposal_id:8, config_seq:9}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}, config_meta={version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}) [2024-04-22 16:23:13.247176] INFO [PALF] receive_config_log (palf_handle_impl.cpp:4067) [11430][T1_L0_G1][T1][YB42AC175882-000616AB08EC2F6B-0-0] [lt=97] receive_config_log success(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", msg_proposal_id=8, prev_lsn={lsn:358454}, prev_log_proposal_id=7, meta={version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}) [2024-04-22 16:23:13.252701] INFO [PALF] append_meta (log_storage.cpp:269) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=84] LogStorage append meta success(ret=0, log_block_header_={magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, log_tail_={lsn:114688}, this={log_tail:{lsn:114688}, readable_log_tail:{lsn:114688}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66990080, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}) [2024-04-22 16:23:13.252931] INFO [PALF] inner_after_flush_meta (palf_handle_impl.cpp:4174) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=64] inner_after_flush_meta(flush_meta_cb_ctx={type:"CHANGE_CONFIG_META", proposal_id:8, config_version:{proposal_id:8, config_seq:9}, base_lsn:{lsn:18446744073709551615}, allow_vote:true, log_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}}) [2024-04-22 16:23:13.253011] INFO [PALF] after_flush_config_log (log_config_mgr.cpp:702) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=69] after_flush_config_log success(palf_id=1, self="172.23.88.129:2882", config_version={proposal_id:8, config_seq:9}, persistent_config_version={proposal_id:8, config_seq:9}) [2024-04-22 16:23:13.253130] INFO [PALF] after_flush_config_change_meta_ (palf_handle_impl.cpp:4262) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=36] submit_change_config_meta_resp success(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, proposal_id=8, config_version={proposal_id:8, config_seq:9}) [2024-04-22 16:23:13.282317] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:42, local_retry_times:42, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.282417] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.282476] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.282496] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.282525] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.286137] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:13.286175] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.286201] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.286216] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.286255] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=37][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.286267] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.286283] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.286351] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.286430] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=68][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.286511] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=77][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.286541] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=43, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.286577] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=43000, remain_us=21540566, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.289875] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:13.289968] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=72] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:13.290000] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:13.290020] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:13.290039] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:13.290057] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:13.290070] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:13.290087] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:13.290099] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:13.290111] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=11][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:13.290123] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=11][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:13.290133] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:13.290161] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=25][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:13.290184] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:13.290198] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.290212] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.290224] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:13.290237] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:13.290253] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:13.290265] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.290291] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:13.290329] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=30][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.290352] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.290363] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:13.290393] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC70-0-0] [lt=13][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:13.290419] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC70-0-0] [lt=21][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:13.290516] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=92][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:13.290587] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:13.290634] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:13.290662] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:13.290712] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd1484050, start=1713774193287730, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:13.290822] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193287680), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:13.292920] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=29][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:13.292987] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=49] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:13.293015] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=25][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:13.293033] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:13.293086] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=50][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:13.293108] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:13.293120] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:13.293137] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:13.293149] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:13.293160] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=10][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:13.293171] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=10][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:13.293181] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:13.293193] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:13.293213] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:13.293226] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.293239] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.293250] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:13.293272] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=18][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:13.293286] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:13.293298] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.293323] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:13.293358] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=27][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.293372] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:13.293381] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:13.293413] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC71-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:13.293488] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC71-0-0] [lt=69][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:13.293541] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:13.293589] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:13.293615] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:13.293684] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:13.293707] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd148e050, start=1713774193290975, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:13.293798] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193290927), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:13.295799] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295833] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295821] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=28][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:13.295847] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295868] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295879] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295872] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=35] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:13.295890] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295901] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295901] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:13.295911] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295924] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295919] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:13.295931] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295937] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295937] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:13.295944] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295957] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295955] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:13.295964] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295967] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:13.295971] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295978] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:13.295983] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:13.295995] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:13.296083] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=84][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:13.296104] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=20][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:13.296121] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=16][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:13.296138] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:13.296159] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:13.296171] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.296185] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:13.296199] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:13.296218] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=15][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193293906), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:21:02.277,"172.23.88.131:2882"|2024-04-22 16:21:02.277]', 'HAPPENED_TIME', '2024-04-22 16:20:58.279862', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:13.296238] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=18][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:13.296256] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=16][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774193293906), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:21:02.277,"172.23.88.131:2882"|2024-04-22 16:21:02.277]', 'HAPPENED_TIME', '2024-04-22 16:20:58.279862', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.296307] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=46][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:13.296354] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC72-0-0] [lt=39][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:13.298373] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-04-22 16:23:13.298905] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60] get wrs ts(ls_id={id:1}, delta=5876521, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.298984] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.299185] INFO [PALF] check_parent_health (log_config_mgr.cpp:2764) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24] re_register_parent reason(palf_id=1, self="172.23.88.129:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713774177861681, last_first_register_time_us=1713773712050886, register_time_us=1713774177861681, curr_time_us=1713774193299181) [2024-04-22 16:23:13.299325] INFO [PALF] check_parent_health (log_config_mgr.cpp:2769) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=46] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:13.329772] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:43, local_retry_times:43, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.329874] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.329901] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.329918] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.329951] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.334066] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] [2024-04-22 16:23:13.334126] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=58][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.334156] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.334178] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.334194] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.334205] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.334217] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.334266] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.334321] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.334342] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.334362] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=44, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.334340] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFBF6-0-0] [lt=105][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774193334235}) [2024-04-22 16:23:13.334398] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=44000, remain_us=21492744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.348099] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] sock regist: 0x7f3cfd8723b0 fd=224 [2024-04-22 16:23:13.348204] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=86] [ussl] accept new connection, fd:224, src_addr:172.23.88.130:55330 [2024-04-22 16:23:13.348384] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=92] [ussl] auth mothod is NONE, the fd will be dispatched, fd:224, src_addr:172.23.88.130:55330 [2024-04-22 16:23:13.348557] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=162] PNIO dispatch fd to certain group, fd:224, gid:0x100000003 [2024-04-22 16:23:13.348670] INFO pkts_sk_init (pkts_sk_factory.h:23) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=18] PNIO set pkts_sk_t sock_id s=0x7f3d01a05078, s->id=327678 [2024-04-22 16:23:13.348727] INFO pkts_sk_new (pkts_sk_factory.h:51) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=55] PNIO sk_new: s=0x7f3d01a05078 [2024-04-22 16:23:13.348809] INFO eloop_regist (eloop.c:47) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO sock regist: 0x7f3d01a05078 fd=224 [2024-04-22 16:23:13.348874] INFO on_accept (listenfd.c:39) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO accept new connection, ns=0x7f3d01a05078, fd=fd:224:local:172.23.88.129:2882:remote:172.23.88.130:55330 [2024-04-22 16:23:13.349057] WDIAG listenfd_handle_event (listenfd.c:71) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:13.349192] WDIAG [PALF] is_prev_log_pid_match (log_sliding_window.cpp:3875) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1E-0-0] [lt=139][errcode=-4018] get_prev_log_info_ failed(tmp_ret=-4018, log_id=2952, palf_id=1, self="172.23.88.129:2882", log_id=2952, lsn={lsn:359848}, prev_lsn={lsn:359726}, prev_log_pid=8) [2024-04-22 16:23:13.349264] WDIAG [PALF] receive_log (log_sliding_window.cpp:3199) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C1E-0-0] [lt=69][errcode=-4023] new log's proposal_id does not equal to last submit log's, and log_id is not continuous with last submit log, cannot receive(cache) it(ret=-4023, palf_id=1, self="172.23.88.129:2882", log_id=2952, log_proposal_id=8, prev_log_proposal_id=8, is_prev_log_exist=false, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359848}, max_scn:{val:1713774197320369684, v:0}, accumulated_checksum:4099157314, log_id:2952, flag:0}, last_submit_log_id=2941, last_submit_log_pid=7) [2024-04-22 16:23:13.378535] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:44, local_retry_times:44, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.378636] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.378665] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.378685] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.378720] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.383545] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] [2024-04-22 16:23:13.383601] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=53][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.383636] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.383665] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.383696] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.383722] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.383764] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.383816] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.383877] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=51][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.383899] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.383923] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=45, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.383962] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21] will sleep(sleep_us=45000, remain_us=21443181, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.397530] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:13.398986] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:13.399048] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:13.399115] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774193398973) [2024-04-22 16:23:13.399149] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774193198943, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:13.399305] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=5976801, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.399366] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.403632] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=91] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:13.406707] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=49] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=9124, clean_start_pos=3900699, clean_num=125829) [2024-04-22 16:23:13.429118] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:45, local_retry_times:45, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.429215] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.429244] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.429258] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.429309] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.432921] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:13.432963] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.432993] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.433016] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.433039] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.433057] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.433070] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.433152] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.433205] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.433226] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.433303] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=73][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=46, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.433384] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63] will sleep(sleep_us=46000, remain_us=21393759, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.479505] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:46, local_retry_times:46, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.479587] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=54][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.479610] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.479625] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.479647] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.483883] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] [2024-04-22 16:23:13.483923] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.483952] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.483974] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.484009] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.484022] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.484035] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.484084] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.484138] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.484159] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.484181] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=47, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.484218] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21] will sleep(sleep_us=47000, remain_us=21342924, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.499127] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] get wrs ts(ls_id={id:1}, delta=6077181, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.499177] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.531319] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:47, local_retry_times:47, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.531418] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.531484] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.531519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.531548] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.535472] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] [2024-04-22 16:23:13.535512] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.535542] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.535565] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.535582] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.535594] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.535605] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.535652] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.535693] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.535706] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.535756] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=48, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.535796] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23] will sleep(sleep_us=48000, remain_us=21291347, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.583928] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:48, local_retry_times:48, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.584073] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=109][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.584097] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.584109] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.584180] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.588324] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [2024-04-22 16:23:13.588365] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.588390] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.588408] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.588489] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=78][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.588519] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.588538] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.588604] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.588662] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.588693] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.588730] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=49, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.588786] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=37] will sleep(sleep_us=49000, remain_us=21238357, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.599216] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:13.599323] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=103][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774193599198) [2024-04-22 16:23:13.599365] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774193399177, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:13.599526] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53] get wrs ts(ls_id={id:1}, delta=6177319, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.599564] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.603791] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:13.607303] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:13.637928] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.637980] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.638003] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.638011] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.638026] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.641245] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] [2024-04-22 16:23:13.641265] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.641276] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.641286] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.641297] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.641305] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.641312] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.641342] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.641365] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.641379] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.641387] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=50, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.641404] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8] will sleep(sleep_us=50000, remain_us=21185737, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.691579] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=52][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:50, local_retry_times:50, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.691702] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=83][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.691731] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.691749] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.691785] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.697336] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=48][errcode=-4725] [2024-04-22 16:23:13.697403] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=65][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.697456] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.697489] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.697505] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.697519] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.697532] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.697583] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.697637] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.697659] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.697680] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=51, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.697717] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22] will sleep(sleep_us=51000, remain_us=21129425, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.699335] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:13.699478] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62] get wrs ts(ls_id={id:1}, delta=6277020, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.699517] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.699734] INFO [SQL.ENG] refresh_statistics (ob_px_tenant_target_monitor.cpp:106) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=17] leader switch to follower(tenant_id_=1, server_="172.23.88.129:2882", leader="172.23.88.130:2882", version_=24924325078906) [2024-04-22 16:23:13.699834] INFO [SQL.ENG] reset_follower_statistics (ob_px_tenant_target_monitor.cpp:367) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=64] reset follower statistics(tenant_id_=1, ret=0, version=18446744073709551615) [2024-04-22 16:23:13.701129] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:13.701410] INFO [SQL.ENG] query_statistics (ob_px_tenant_target_monitor.cpp:290) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=22] monitor version not match(tenant_id_=1, leader_version=24924333433916, version_=18446744073709551615) [2024-04-22 16:23:13.701537] INFO [SQL.ENG] reset_follower_statistics (ob_px_tenant_target_monitor.cpp:367) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=93] reset follower statistics(tenant_id_=1, ret=0, version=24924333433916) [2024-04-22 16:23:13.702235] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [10458][pnio1][T0][YB42AC175881-000616AB0B5FEDC9-0-0] [lt=33] ts response task statistics(alloc_count=3606, free_count=1802) [2024-04-22 16:23:13.704645] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=110] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:13.741730] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3582-0-0] [lt=233][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:13.741825] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3582-0-0] [lt=92][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:13.741869] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3582-0-0] [lt=40][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:13.741904] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3582-0-0] [lt=32][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:13.748971] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=53][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:51, local_retry_times:51, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.749057] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.749080] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.749091] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.749113] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.754655] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] [2024-04-22 16:23:13.754690] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.754716] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.754735] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.754753] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.754787] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.754802] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.754849] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.754882] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.754893] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.754909] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=52, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.754941] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15] will sleep(sleep_us=52000, remain_us=21072201, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.799431] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=73][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:13.799604] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=133][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774187421454783, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774193799412) [2024-04-22 16:23:13.799637] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774193599419, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:13.799702] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=6377815, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.799752] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.799817] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:126) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] wead read timestamp is too old(timestamp={val:1713774187421454783, v:0}, gts_scn={val:1713774197353219954, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1713774187421454783, v:0}}) [2024-04-22 16:23:13.799899] INFO [STORAGE.TRANS] amend (ob_tenant_weak_read_server_version_mgr.cpp:192) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63] [WRS] tenant has no valid partition on current server, generate weak read snapshot version based on current timestamp(tenant_id=1, epoch_tstamp_=1713774193799661, total_part_count_=1, valid_inner_part_count_=0, valid_user_part_count_=0, version_={val:1713774192853219000, v:0}, base_version_when_no_valid_partition={val:1713774192853219000, v:0}) [2024-04-22 16:23:13.803961] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:13.807085] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:52, local_retry_times:52, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.807213] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.807240] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.807295] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=52][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.807332] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.807814] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:13.811186] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] [2024-04-22 16:23:13.811242] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=53][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.811274] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.811298] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.811316] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.811327] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.811339] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.811389] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.811480] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=43][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.811505] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.811588] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=76][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=53, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.811652] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40] will sleep(sleep_us=53000, remain_us=21015491, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.864862] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:53, local_retry_times:53, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.864990] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=85][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.865019] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.865038] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.865075] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.869237] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] [2024-04-22 16:23:13.869290] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.869324] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.869348] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.869389] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.869402] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:13.869413] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.869506] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.869565] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:13.869588] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:13.869610] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=54, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:13.869647] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=54000, remain_us=20957495, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.899633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=6478069, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.899709] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:13.923795] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:54, local_retry_times:54, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:13.923960] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=83][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.923986] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:13.924022] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:13.924053] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:13.928655] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:13.928712] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=54][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:13.928743] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:13.928761] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:13.928776] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:13.928787] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:13.928845] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.928974] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=55000, remain_us=20898169, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.989422] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:13.989596] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=56000, remain_us=20837547, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:13.999690] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:13.999760] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774192853219000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1713774193999676) [2024-04-22 16:23:13.999798] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774193799659, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:13.999871] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=6577461, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:13.999945] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:14.004199] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=78] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:14.008491] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:14.036273] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [10835][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=28] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f3cec9dc2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-22 16:23:14.051254] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.051356] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=57000, remain_us=20775787, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.099817] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] get wrs ts(ls_id={id:1}, delta=6677872, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:14.099880] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:14.112846] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.112942] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=58000, remain_us=20714200, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.175296] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.175392] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=59000, remain_us=20651751, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=59, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.189807] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.190055] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.190327] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=64] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.190390] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.190719] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.190851] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=74] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.191194] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.191315] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.191642] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.191849] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=47] gc stale ls task succ [2024-04-22 16:23:14.191948] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.192032] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10455, mem_used=79030272) [2024-04-22 16:23:14.192073] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.192182] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=81] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.192198] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=42] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:14.192542] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=63] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.192813] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:14.192859] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.192858] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=13] ====== check clog disk timer task ====== [2024-04-22 16:23:14.192910] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=36] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:14.192976] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=28] cannot_recycle_log_size statistics(cannot_recycle_log_size=358576, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:14.193581] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=110] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.193698] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=70] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.194130] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.194605] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.194724] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=27] current monitor number(seq_=-1) [2024-04-22 16:23:14.194958] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:14.195018] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:14.195072] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:14.195095] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:14.195918] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.196810] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=71] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.196809] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=52] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.197812] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.197829] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.197824] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=34] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:14.198634] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:12, tid:10410}]) [2024-04-22 16:23:14.199580] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:14.199902] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:14.199971] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774192853219000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1713774194199888) [2024-04-22 16:23:14.200001] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774193999822, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:14.200088] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=6777638, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:14.200154] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:14.200970] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=10][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:14.201046] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:14.201079] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:14.201100] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:14.201119] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:14.201137] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:14.201150] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:14.201191] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=32][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:14.201204] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:14.201216] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:14.201227] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:14.201238] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:14.201250] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:14.201262] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:14.201274] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:14.201301] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:14.201323] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=17][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:14.201364] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=34][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:14.201388] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=22][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:14.201408] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=17][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:14.201502] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=88][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:14.201535] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=28][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.201578] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=35][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:14.201614] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=28][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:14.201647] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=30][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:14.201660] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=12][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:14.201685] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED81-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:14.201663] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=18] PNIO [ratelimit] time: 1713774194201659, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007591, add_bytes: 0 [2024-04-22 16:23:14.201725] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=35][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:14.201781] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=53][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:14.201796] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:14.201822] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=25][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:14.201837] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd227a050, start=1713774194200731, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:14.201857] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=18][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:14.201871] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=10][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:14.201984] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=13][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:14.202008] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED81-0-0] [lt=22][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:14.202573] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED70-0-0] [lt=96] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, location:{server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:8}}]) [2024-04-22 16:23:14.202647] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=69] PNIO [ratelimit] time: 1713774194202643, bytes: 66131706, bw: 0.019435 MB/s, add_ts: 1002759, add_bytes: 20435 [2024-04-22 16:23:14.203133] WDIAG [SQL.ENG] update_peer_target_used (ob_px_tenant_target_monitor.cpp:332) [11015][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4201] get refactored failed(ret=-4201, tenant_id_=1, server="172.23.88.130:2882", version_=24924333433916) [2024-04-22 16:23:14.203408] WDIAG [RPC] send (ob_poc_rpc_proxy.h:173) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4638] execute rpc fail(addr="172.23.88.129:2882", pcode=258, ret=-4638, timeout=2000000) [2024-04-22 16:23:14.203505] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:244) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=91][errcode=-4638] [2024-04-22 16:23:14.203841] INFO pktc_sk_new (pktc_sk_factory.h:78) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO sk_new: s=0x7f3cd332c048 [2024-04-22 16:23:14.203951] INFO pktc_sk_new (pktc_sk_factory.h:78) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=40] PNIO sk_new: s=0x7f3d01bf8048 [2024-04-22 16:23:14.204158] INFO pktc_do_connect (pktc_post.h:19) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=46] PNIO sk_new: sk=0x7f3cd332c048, fd=225 [2024-04-22 16:23:14.204200] INFO pktc_do_connect (pktc_post.h:19) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=56] PNIO sk_new: sk=0x7f3d01bf8048, fd=229 [2024-04-22 16:23:14.204209] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] write client fd succ, fd:225, gid:0x100000002, need_send_negotiation:1 [2024-04-22 16:23:14.204240] INFO eloop_regist (eloop.c:47) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO sock regist: 0x7f3cd332c048 fd=225 [2024-04-22 16:23:14.204236] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] write client fd succ, fd:229, gid:0x100000003, need_send_negotiation:1 [2024-04-22 16:23:14.204252] INFO eloop_regist (eloop.c:47) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO sock regist: 0x7f3d01bf8048 fd=229 [2024-04-22 16:23:14.204252] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] sock regist: 0x7f3cfd8723b0 fd=231 [2024-04-22 16:23:14.204277] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO sock not ready: 0x7f3cd332c048, fd=225 [2024-04-22 16:23:14.204284] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO sock not ready: 0x7f3d01bf8048, fd=229 [2024-04-22 16:23:14.204295] INFO ussl_on_accept (ussl_listenfd.c:40) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=28] [ussl] accept new connection, fd:231, src_addr:172.23.88.129:33162 [2024-04-22 16:23:14.204368] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] sock regist: 0x7f3cfd872040 fd=225 [2024-04-22 16:23:14.204424] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] sock regist: 0x7f3cfd8074d0 fd=229 [2024-04-22 16:23:14.204493] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=77] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:14.204568] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=65] [ussl] client send negotiation message succ, fd:229, addr:172.23.88.129:59744, auth_method:NONE, gid:0x100000003 [2024-04-22 16:23:14.204638] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] give back fd to origin epoll succ, client_fd:229, client_epfd:124, event:0x8000000d, client_addr:172.23.88.131:2882, has_error:0 [2024-04-22 16:23:14.204708] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO sock connect OK: 0x7f3d01bf8048 fd:229:local:172.23.88.129:59744:remote:172.23.88.131:2882 [2024-04-22 16:23:14.204742] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=31] [ussl] client send negotiation message succ, fd:225, addr:172.23.88.129:33162, auth_method:NONE, gid:0x100000002 [2024-04-22 16:23:14.204790] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] give back fd to origin epoll succ, client_fd:225, client_epfd:117, event:0x8000000d, client_addr:172.23.88.129:2882, has_error:0 [2024-04-22 16:23:14.204857] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sock connect OK: 0x7f3cd332c048 fd:225:local:172.23.88.129:33162:remote:172.23.88.129:2882 [2024-04-22 16:23:14.204881] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=49] [ussl] auth mothod is NONE, the fd will be dispatched, fd:231, src_addr:172.23.88.129:33162 [2024-04-22 16:23:14.204903] INFO dispatch_accept_fd_to_certain_group (group.c:642) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:231, gid:0x100000002 [2024-04-22 16:23:14.204967] INFO pkts_sk_init (pkts_sk_factory.h:23) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO set pkts_sk_t sock_id s=0x7f3cd332c9e8, s->id=589821 [2024-04-22 16:23:14.205002] INFO pkts_sk_new (pkts_sk_factory.h:51) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=34] PNIO sk_new: s=0x7f3cd332c9e8 [2024-04-22 16:23:14.205073] INFO eloop_regist (eloop.c:47) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO sock regist: 0x7f3cd332c9e8 fd=231 [2024-04-22 16:23:14.205120] INFO on_accept (listenfd.c:39) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO accept new connection, ns=0x7f3cd332c9e8, fd=fd:231:local:172.23.88.129:2882:remote:172.23.88.129:33162 [2024-04-22 16:23:14.205234] WDIAG listenfd_handle_event (listenfd.c:71) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:23:14.205976] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=23] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774191670314, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:23:14.206100] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=7] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774191670314, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:23:14.206180] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=196] [RS_MGR] new master rootserver found(rootservice="172.23.88.130:2882", cluster_id=1713772869) [2024-04-22 16:23:14.206230] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:582) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=94] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2024-04-22 16:23:14.206278] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:221) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=24] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=307) [2024-04-22 16:23:14.207611] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=34][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:14.207665] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=40] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:14.207692] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:14.207727] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=33][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:14.207749] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=20][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:14.207769] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:14.207782] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:14.207801] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:14.207814] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:14.207827] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:14.207836] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=8][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:14.207844] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:14.207853] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:14.207862] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:14.207870] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:14.207857] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:14.207898] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=19][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:14.207908] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:14.207919] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:14.207928] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:14.207937] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:14.207960] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=21][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:14.207970] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.208020] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:14.208026] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=36] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:14.208052] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=26][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:14.208062] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:14.208054] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:23:14.208070] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:14.208087] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:14.208112] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=23][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:14.208126] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:14.208137] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:14.208145] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:14.208181] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45cc050, start=1713774194207408, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:14.208226] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=42][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:14.208248] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=19][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:14.208130] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:261) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:14.208335] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:305) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=178] [REFRESH_SCHEMA] begin refresh schema, (begin_time=1713774194208332) [2024-04-22 16:23:14.208362] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2388) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[]) [2024-04-22 16:23:14.208395] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=23][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:14.208411] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=14][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:14.208423] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:14.208459] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=8][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:14.208484] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=21][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:14.208497] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=11][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:14.208506] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7D-0-0] [lt=9][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:14.209185] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=104] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20730150912, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:14.213653] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:14.213794] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.213882] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] already timeout, do not need sleep(sleep_us=0, remain_us=1994527, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.217585] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFD0D-0-0] [lt=86][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774193201832}) [2024-04-22 16:23:14.217663] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.217798] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=1000, remain_us=1990612, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.223624] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.223717] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=2000, remain_us=1984692, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.229926] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.230012] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=3000, remain_us=1978397, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.237169] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.237266] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=4000, remain_us=1971144, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.238746] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3585-0-0] [lt=155][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:14.238816] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3585-0-0] [lt=66][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:14.238865] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3585-0-0] [lt=46][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:14.238901] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3585-0-0] [lt=35][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:14.239501] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.239612] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=60000, remain_us=20587530, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=60, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.245153] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.245251] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=5000, remain_us=1963159, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.250706] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=39] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:14.251864] WDIAG [PALF] is_prev_log_pid_match (log_sliding_window.cpp:3875) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C20-0-0] [lt=52][errcode=-4018] get_prev_log_info_ failed(tmp_ret=-4018, log_id=2953, palf_id=1, self="172.23.88.129:2882", log_id=2953, lsn={lsn:359970}, prev_lsn={lsn:359848}, prev_log_pid=8) [2024-04-22 16:23:14.251934] WDIAG [PALF] receive_log (log_sliding_window.cpp:3199) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C20-0-0] [lt=67][errcode=-4023] new log's proposal_id does not equal to last submit log's, and log_id is not continuous with last submit log, cannot receive(cache) it(ret=-4023, palf_id=1, self="172.23.88.129:2882", log_id=2953, log_proposal_id=8, prev_log_proposal_id=8, is_prev_log_exist=false, group_entry_header={magic:18258, version:1, group_size:57, proposal_id:8, committed_lsn:{lsn:359970}, max_scn:{val:1713774197406167741, v:0}, accumulated_checksum:295676114, log_id:2953, flag:0}, last_submit_log_id=2941, last_submit_log_pid=7) [2024-04-22 16:23:14.254208] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.254295] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=6000, remain_us=1954114, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.264190] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.264278] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=7000, remain_us=1944131, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.274911] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.275025] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1933385, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.287429] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.287588] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=1920821, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.297547] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297615] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=65] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297644] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297666] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297682] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297695] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297709] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297722] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297791] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=66] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297823] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297891] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=64] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297909] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297923] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297937] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297950] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.297964] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:14.299114] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=84][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:230, tid:10796}]) [2024-04-22 16:23:14.299741] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:60, local_retry_times:60, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.299900] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=127][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.299930] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.299945] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.299972] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.300025] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=90] get wrs ts(ls_id={id:1}, delta=6877830, log_service_ts=1713774187421454, min_tx_service_ts=4611686018427387, timestamp={val:1713774187421454783, v:0}) [2024-04-22 16:23:14.300086] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:14.300474] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2107) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=43] do_fetch_log_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", dest="172.23.88.130:2882", fetch_log_count=2048, fetch_start_lsn={lsn:358576}, prev_lsn={lsn:358454}, fetch_start_log_id=2942, last_slide_log_id=2941, fetch_log_size=41943040, accepted_mode_pid=8, last_fetch_req_time=1713774194300468, last_fetch_end_lsn={lsn:42301616}, last_fetch_max_log_id=4989, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.23.88.129:2882", lsn_allocator:{max_log_id:2941, max_lsn:358576, max_scn:{val:1713774187421454783, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:358576}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:358576}}, last_submit_lsn:{lsn:358454}, last_submit_end_lsn:{lsn:358576}, last_submit_log_id:2941, last_submit_log_pid:7, max_flushed_lsn:{lsn:358454}, max_flushed_end_lsn:{lsn:358576}, max_flushed_log_pid:7, committed_end_lsn:{lsn:358576}, last_slide_log_id:2941, last_slide_scn:{val:1713774187421454783, v:0}, last_slide_lsn:{lsn:358454}, last_slide_end_lsn:{lsn:358576}, last_slide_log_pid:7, last_slide_log_accum_checksum:1087531090, last_fetch_end_lsn:{lsn:42301616}, last_fetch_max_log_id:4989, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713774194300468, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f3cd2de4530}) [2024-04-22 16:23:14.300830] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0][errcode=-4725] [2024-04-22 16:23:14.300874] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.300901] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.300914] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.300925] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.300934] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.300942] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.300980] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.301083] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=95][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.301107] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.301143] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=10, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.301172] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17] will sleep(sleep_us=10000, remain_us=1907237, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.301998] INFO [PALF] receive_log (log_sliding_window.cpp:3225) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=46] new log's proposal_id does not equal to last submit log's, and it is continuous, need clean cached log_tasks beyond this log(ret=-4023, palf_id=1, self="172.23.88.129:2882", log_id=2942, log_proposal_id=8, is_prev_log_exist=true, group_entry_header={magic:18258, version:1, group_size:112, proposal_id:8, committed_lsn:{lsn:358576}, max_scn:{val:1713774187421454784, v:0}, accumulated_checksum:3241726231, log_id:2942, flag:1}, last_submit_log_id=2941, last_submit_log_pid=7) [2024-04-22 16:23:14.302089] INFO [PALF] receive_log_ (palf_handle_impl.cpp:3066) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=87] begin clean/truncate log(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", self="172.23.88.129:2882", msg_proposal_id=8, lsn={lsn:358576}, buf_len=168, truncate_log_info={truncate_type:"TRUNCATE_CACHED_LOG_TASK", truncate_log_id:2942, truncate_begin_lsn:{lsn:18446744073709551615}, truncate_log_proposal_id:9223372036854775807}) [2024-04-22 16:23:14.302161] INFO [PALF] clean_log_ (log_sliding_window.cpp:2514) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=48] record last slide log info(ret=0, last_slide_log_id=2941, last_slide_scn={val:1713774187421454783, v:0}, last_slide_end_lsn={lsn:358576}, palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:14.302224] INFO [PALF] truncate (lsn_allocator.cpp:93) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=38] truncate success(lsn={lsn:358576}, log_id=2941, scn={val:1713774187421454783, v:0}) [2024-04-22 16:23:14.302255] INFO [PALF] truncate_lsn_allocator_ (log_sliding_window.cpp:4366) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=29] lsn_allocator_.truncate success(ret=0, palf_id=1, self="172.23.88.129:2882", last_lsn={lsn:358576}, last_log_id=2941, last_scn={val:1713774187421454783, v:0}) [2024-04-22 16:23:14.302341] INFO [PALF] clean_log_ (log_sliding_window.cpp:2576) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=82] clean log finished(ret=0, palf_id=1, self="172.23.88.129:2882", max_log_id=2941, first_empty_log_id=-1, last_submit_log_id=2941, start_log_id=2942, max_log_id=2941, new_last_log_id=2941, new_last_scn={val:1713774187421454783, v:0}, new_last_log_end_lsn={lsn:358576}) [2024-04-22 16:23:14.302385] INFO [PALF] clean_cached_log (log_sliding_window.cpp:2467) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=42] clean_cached_log finished(ret=0, palf_id=1, self="172.23.88.129:2882", begin_log_id=2942, lsn={lsn:358576}, prev_lsn={lsn:358454}, prev_log_pid=7) [2024-04-22 16:23:14.302485] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=31] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:14.302558] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=1, avg_log_batch_cnt=1, total_group_log_size=290, avg_group_log_size=290) [2024-04-22 16:23:14.302617] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=51] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=122) [2024-04-22 16:23:14.302731] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2107) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=25] do_fetch_log_ finished(ret=0, palf_id=1, self="172.23.88.129:2882", dest="172.23.88.130:2882", fetch_log_count=2047, fetch_start_lsn={lsn:358744}, prev_lsn={lsn:358576}, fetch_start_log_id=2943, last_slide_log_id=2941, fetch_log_size=41943040, accepted_mode_pid=8, last_fetch_req_time=1713774194302728, last_fetch_end_lsn={lsn:42301784}, last_fetch_max_log_id=4989, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="CLEAN_CACHED_LOG", this={palf_id:1, self:"172.23.88.129:2882", lsn_allocator:{max_log_id:2942, max_lsn:358744, max_scn:{val:1713774187421454784, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:358576}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:358576}}, last_submit_lsn:{lsn:358576}, last_submit_end_lsn:{lsn:358744}, last_submit_log_id:2942, last_submit_log_pid:8, max_flushed_lsn:{lsn:358454}, max_flushed_end_lsn:{lsn:358576}, max_flushed_log_pid:7, committed_end_lsn:{lsn:358576}, last_slide_log_id:2941, last_slide_scn:{val:1713774187421454783, v:0}, last_slide_lsn:{lsn:358454}, last_slide_end_lsn:{lsn:358576}, last_slide_log_pid:7, last_slide_log_accum_checksum:1087531090, last_fetch_end_lsn:{lsn:42301784}, last_fetch_max_log_id:4989, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1713774194302728, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"CLEAN_CACHED_LOG", this:0x7f3cd2de4530}) [2024-04-22 16:23:14.302883] INFO [PALF] receive_log_ (palf_handle_impl.cpp:3103) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F19-0-0] [lt=148] receive_log success(ret=0, this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", self="172.23.88.129:2882", msg_proposal_id=8, prev_lsn={lsn:358454}, prev_log_proposal_id=7, lsn={lsn:358576}, truncate_log_info={truncate_type:"TRUNCATE_CACHED_LOG_TASK", truncate_log_id:2942, truncate_begin_lsn:{lsn:18446744073709551615}, truncate_log_proposal_id:9223372036854775807}) [2024-04-22 16:23:14.303215] INFO [PALF] after_flush_mode_meta (log_mode_mgr.cpp:771) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1A-0-0] [lt=38] after_flush_mode_meta success(ret=0, palf_id=1, self="172.23.88.129:2882", is_applied_mode_meta=true, mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, applied_mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:14.303274] INFO [PALF] receive_mode_meta (palf_handle_impl.cpp:2793) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1A-0-0] [lt=56] receive_mode_meta finish(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", proposal_id=8, is_applied_mode_meta=true, mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:14.303305] INFO [PALF] handle_request (log_request_handler.cpp:335) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1A-0-0] [lt=29] receive_mode_meta success(ret=0, palf_id=1, server="172.23.88.130:2882", req={msg_proposal_id:8, meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:true}, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:12, flying_meta_task:0}}) [2024-04-22 16:23:14.304145] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] [2024-04-22 16:23:14.304203] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.304236] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.304256] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.304271] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.304283] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.304294] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.304347] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.304434] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=79][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.304395] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=71] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2943, log_end_lsn={lsn:358866}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:358744}, end_lsn:{lsn:358866}, log_id:2943, min_scn:{val:1713774187462453100, v:0}, max_scn:{val:1713774187462453100, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358576}, prev_proposal_id:8, committed_end_lsn:{lsn:358744}, data_checksum:667309872, accum_checksum:4217822272, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302935, freeze_ts:1713774194302935, submit_ts:1713774194302942, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:358744}, max_scn:{val:1713774187462453100, v:0}, accumulated_checksum:4217822272, log_id:2943, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:358744}, end_lsn:{lsn:358866}, log_id:2943, min_scn:{val:1713774187462453100, v:0}, max_scn:{val:1713774187462453100, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358576}, prev_proposal_id:8, committed_end_lsn:{lsn:358744}, data_checksum:667309872, accum_checksum:4217822272, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302935, freeze_ts:1713774194302935, submit_ts:1713774194302942, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:0}) [2024-04-22 16:23:14.304518] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=80][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:14.304545] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=147] no need update log(log_id=2943, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:358744}, max_scn:{val:1713774187462453100, v:0}, accumulated_checksum:4217822272, log_id:2943, flag:0}, log_task={header:{begin_lsn:{lsn:358744}, end_lsn:{lsn:358866}, log_id:2943, min_scn:{val:1713774187462453100, v:0}, max_scn:{val:1713774187462453100, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358576}, prev_proposal_id:8, committed_end_lsn:{lsn:358744}, data_checksum:667309872, accum_checksum:4217822272, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302935, freeze_ts:1713774194302935, submit_ts:1713774194302942, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:0}) [2024-04-22 16:23:14.304604] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=83][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=61, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.304648] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27] will sleep(sleep_us=61000, remain_us=20522494, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=61, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.304617] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=60] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2944, log_end_lsn={lsn:358988}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:358866}, end_lsn:{lsn:358988}, log_id:2944, min_scn:{val:1713774187462453101, v:0}, max_scn:{val:1713774187462453101, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358744}, prev_proposal_id:8, committed_end_lsn:{lsn:358866}, data_checksum:667309872, accum_checksum:4198493115, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302954, freeze_ts:1713774194302954, submit_ts:1713774194302957, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:358866}, max_scn:{val:1713774187462453101, v:0}, accumulated_checksum:4198493115, log_id:2944, flag:1}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:358866}, end_lsn:{lsn:358988}, log_id:2944, min_scn:{val:1713774187462453101, v:0}, max_scn:{val:1713774187462453101, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358744}, prev_proposal_id:8, committed_end_lsn:{lsn:358866}, data_checksum:667309872, accum_checksum:4198493115, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302954, freeze_ts:1713774194302954, submit_ts:1713774194302957, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.304700] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=81] no need update log(log_id=2944, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:358866}, max_scn:{val:1713774187462453101, v:0}, accumulated_checksum:4198493115, log_id:2944, flag:1}, log_task={header:{begin_lsn:{lsn:358866}, end_lsn:{lsn:358988}, log_id:2944, min_scn:{val:1713774187462453101, v:0}, max_scn:{val:1713774187462453101, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358744}, prev_proposal_id:8, committed_end_lsn:{lsn:358866}, data_checksum:667309872, accum_checksum:4198493115, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302954, freeze_ts:1713774194302954, submit_ts:1713774194302957, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.304793] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=84] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2945, log_end_lsn={lsn:359110}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:358988}, end_lsn:{lsn:359110}, log_id:2945, min_scn:{val:1713774187462453102, v:0}, max_scn:{val:1713774187462453102, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358866}, prev_proposal_id:8, committed_end_lsn:{lsn:358988}, data_checksum:667309872, accum_checksum:33417118, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302965, freeze_ts:1713774194302965, submit_ts:1713774194302968, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:358988}, max_scn:{val:1713774187462453102, v:0}, accumulated_checksum:33417118, log_id:2945, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:358988}, end_lsn:{lsn:359110}, log_id:2945, min_scn:{val:1713774187462453102, v:0}, max_scn:{val:1713774187462453102, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358866}, prev_proposal_id:8, committed_end_lsn:{lsn:358988}, data_checksum:667309872, accum_checksum:33417118, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302965, freeze_ts:1713774194302965, submit_ts:1713774194302968, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.304888] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=93] no need update log(log_id=2945, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:358988}, max_scn:{val:1713774187462453102, v:0}, accumulated_checksum:33417118, log_id:2945, flag:0}, log_task={header:{begin_lsn:{lsn:358988}, end_lsn:{lsn:359110}, log_id:2945, min_scn:{val:1713774187462453102, v:0}, max_scn:{val:1713774187462453102, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:358866}, prev_proposal_id:8, committed_end_lsn:{lsn:358988}, data_checksum:667309872, accum_checksum:33417118, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302965, freeze_ts:1713774194302965, submit_ts:1713774194302968, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.304947] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=51] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2946, log_end_lsn={lsn:359238}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:359110}, end_lsn:{lsn:359238}, log_id:2946, min_scn:{val:1713774197210958129, v:0}, max_scn:{val:1713774197210958129, v:0}, data_len:72, proposal_id:8, prev_lsn:{lsn:358988}, prev_proposal_id:8, committed_end_lsn:{lsn:359110}, data_checksum:2214260913, accum_checksum:2887112897, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302991, freeze_ts:1713774194302991, submit_ts:1713774194302995, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:72, proposal_id:8, committed_lsn:{lsn:359110}, max_scn:{val:1713774197210958129, v:0}, accumulated_checksum:2887112897, log_id:2946, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:359110}, end_lsn:{lsn:359238}, log_id:2946, min_scn:{val:1713774197210958129, v:0}, max_scn:{val:1713774197210958129, v:0}, data_len:72, proposal_id:8, prev_lsn:{lsn:358988}, prev_proposal_id:8, committed_end_lsn:{lsn:359110}, data_checksum:2214260913, accum_checksum:2887112897, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302991, freeze_ts:1713774194302991, submit_ts:1713774194302995, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.305043] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=95] no need update log(log_id=2946, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:72, proposal_id:8, committed_lsn:{lsn:359110}, max_scn:{val:1713774197210958129, v:0}, accumulated_checksum:2887112897, log_id:2946, flag:0}, log_task={header:{begin_lsn:{lsn:359110}, end_lsn:{lsn:359238}, log_id:2946, min_scn:{val:1713774197210958129, v:0}, max_scn:{val:1713774197210958129, v:0}, data_len:72, proposal_id:8, prev_lsn:{lsn:358988}, prev_proposal_id:8, committed_end_lsn:{lsn:359110}, data_checksum:2214260913, accum_checksum:2887112897, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194302991, freeze_ts:1713774194302991, submit_ts:1713774194302995, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.305133] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=76] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2947, log_end_lsn={lsn:359360}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:359238}, end_lsn:{lsn:359360}, log_id:2947, min_scn:{val:1713774197210958130, v:0}, max_scn:{val:1713774197210958130, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359110}, prev_proposal_id:8, committed_end_lsn:{lsn:359110}, data_checksum:667309872, accum_checksum:1302820498, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303001, freeze_ts:1713774194303001, submit_ts:1713774194303005, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359110}, max_scn:{val:1713774197210958130, v:0}, accumulated_checksum:1302820498, log_id:2947, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:359238}, end_lsn:{lsn:359360}, log_id:2947, min_scn:{val:1713774197210958130, v:0}, max_scn:{val:1713774197210958130, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359110}, prev_proposal_id:8, committed_end_lsn:{lsn:359110}, data_checksum:667309872, accum_checksum:1302820498, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303001, freeze_ts:1713774194303001, submit_ts:1713774194303005, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.305285] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=150] no need update log(log_id=2947, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359110}, max_scn:{val:1713774197210958130, v:0}, accumulated_checksum:1302820498, log_id:2947, flag:0}, log_task={header:{begin_lsn:{lsn:359238}, end_lsn:{lsn:359360}, log_id:2947, min_scn:{val:1713774197210958130, v:0}, max_scn:{val:1713774197210958130, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359110}, prev_proposal_id:8, committed_end_lsn:{lsn:359110}, data_checksum:667309872, accum_checksum:1302820498, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303001, freeze_ts:1713774194303001, submit_ts:1713774194303005, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.305366] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=68] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2948, log_end_lsn={lsn:359482}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:359360}, end_lsn:{lsn:359482}, log_id:2948, min_scn:{val:1713774197210958131, v:0}, max_scn:{val:1713774197210958131, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359238}, prev_proposal_id:8, committed_end_lsn:{lsn:359360}, data_checksum:667309872, accum_checksum:56462217, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303012, freeze_ts:1713774194303012, submit_ts:1713774194303016, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359360}, max_scn:{val:1713774197210958131, v:0}, accumulated_checksum:56462217, log_id:2948, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:359360}, end_lsn:{lsn:359482}, log_id:2948, min_scn:{val:1713774197210958131, v:0}, max_scn:{val:1713774197210958131, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359238}, prev_proposal_id:8, committed_end_lsn:{lsn:359360}, data_checksum:667309872, accum_checksum:56462217, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303012, freeze_ts:1713774194303012, submit_ts:1713774194303016, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.305547] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=179] no need update log(log_id=2948, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359360}, max_scn:{val:1713774197210958131, v:0}, accumulated_checksum:56462217, log_id:2948, flag:0}, log_task={header:{begin_lsn:{lsn:359360}, end_lsn:{lsn:359482}, log_id:2948, min_scn:{val:1713774197210958131, v:0}, max_scn:{val:1713774197210958131, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359238}, prev_proposal_id:8, committed_end_lsn:{lsn:359360}, data_checksum:667309872, accum_checksum:56462217, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303012, freeze_ts:1713774194303012, submit_ts:1713774194303016, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.305642] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=83] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2949, log_end_lsn={lsn:359604}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:359482}, end_lsn:{lsn:359604}, log_id:2949, min_scn:{val:1713774197210958132, v:0}, max_scn:{val:1713774197210958132, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359360}, prev_proposal_id:8, committed_end_lsn:{lsn:359482}, data_checksum:667309872, accum_checksum:1030494164, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303023, freeze_ts:1713774194303023, submit_ts:1713774194303026, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359482}, max_scn:{val:1713774197210958132, v:0}, accumulated_checksum:1030494164, log_id:2949, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:359482}, end_lsn:{lsn:359604}, log_id:2949, min_scn:{val:1713774197210958132, v:0}, max_scn:{val:1713774197210958132, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359360}, prev_proposal_id:8, committed_end_lsn:{lsn:359482}, data_checksum:667309872, accum_checksum:1030494164, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303023, freeze_ts:1713774194303023, submit_ts:1713774194303026, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.305766] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=91] no need update log(log_id=2949, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359482}, max_scn:{val:1713774197210958132, v:0}, accumulated_checksum:1030494164, log_id:2949, flag:0}, log_task={header:{begin_lsn:{lsn:359482}, end_lsn:{lsn:359604}, log_id:2949, min_scn:{val:1713774197210958132, v:0}, max_scn:{val:1713774197210958132, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359360}, prev_proposal_id:8, committed_end_lsn:{lsn:359482}, data_checksum:667309872, accum_checksum:1030494164, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303023, freeze_ts:1713774194303023, submit_ts:1713774194303026, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.305851] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=72] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2950, log_end_lsn={lsn:359726}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:359604}, end_lsn:{lsn:359726}, log_id:2950, min_scn:{val:1713774197210958133, v:0}, max_scn:{val:1713774197210958133, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359482}, prev_proposal_id:8, committed_end_lsn:{lsn:359604}, data_checksum:667309872, accum_checksum:1622806853, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303033, freeze_ts:1713774194303033, submit_ts:1713774194303037, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359604}, max_scn:{val:1713774197210958133, v:0}, accumulated_checksum:1622806853, log_id:2950, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:359604}, end_lsn:{lsn:359726}, log_id:2950, min_scn:{val:1713774197210958133, v:0}, max_scn:{val:1713774197210958133, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359482}, prev_proposal_id:8, committed_end_lsn:{lsn:359604}, data_checksum:667309872, accum_checksum:1622806853, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303033, freeze_ts:1713774194303033, submit_ts:1713774194303037, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.306004] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=151] no need update log(log_id=2950, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359604}, max_scn:{val:1713774197210958133, v:0}, accumulated_checksum:1622806853, log_id:2950, flag:0}, log_task={header:{begin_lsn:{lsn:359604}, end_lsn:{lsn:359726}, log_id:2950, min_scn:{val:1713774197210958133, v:0}, max_scn:{val:1713774197210958133, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359482}, prev_proposal_id:8, committed_end_lsn:{lsn:359604}, data_checksum:667309872, accum_checksum:1622806853, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303033, freeze_ts:1713774194303033, submit_ts:1713774194303037, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.306066] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=53] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2951, log_end_lsn={lsn:359848}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:359726}, end_lsn:{lsn:359848}, log_id:2951, min_scn:{val:1713774197210958134, v:0}, max_scn:{val:1713774197210958134, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359604}, prev_proposal_id:8, committed_end_lsn:{lsn:359726}, data_checksum:667309872, accum_checksum:4063163828, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303043, freeze_ts:1713774194303043, submit_ts:1713774194303047, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359726}, max_scn:{val:1713774197210958134, v:0}, accumulated_checksum:4063163828, log_id:2951, flag:1}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:359726}, end_lsn:{lsn:359848}, log_id:2951, min_scn:{val:1713774197210958134, v:0}, max_scn:{val:1713774197210958134, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359604}, prev_proposal_id:8, committed_end_lsn:{lsn:359726}, data_checksum:667309872, accum_checksum:4063163828, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303043, freeze_ts:1713774194303043, submit_ts:1713774194303047, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.306168] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=100] no need update log(log_id=2951, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359726}, max_scn:{val:1713774197210958134, v:0}, accumulated_checksum:4063163828, log_id:2951, flag:1}, log_task={header:{begin_lsn:{lsn:359726}, end_lsn:{lsn:359848}, log_id:2951, min_scn:{val:1713774197210958134, v:0}, max_scn:{val:1713774197210958134, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359604}, prev_proposal_id:8, committed_end_lsn:{lsn:359726}, data_checksum:667309872, accum_checksum:4063163828, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303043, freeze_ts:1713774194303043, submit_ts:1713774194303047, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:4, submit_to_flush cost time:0}) [2024-04-22 16:23:14.306235] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=58] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2952, log_end_lsn={lsn:359970}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:359848}, end_lsn:{lsn:359970}, log_id:2952, min_scn:{val:1713774197320369684, v:0}, max_scn:{val:1713774197320369684, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359726}, prev_proposal_id:8, committed_end_lsn:{lsn:359848}, data_checksum:667309872, accum_checksum:4099157314, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303053, freeze_ts:1713774194303053, submit_ts:1713774194303056, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359848}, max_scn:{val:1713774197320369684, v:0}, accumulated_checksum:4099157314, log_id:2952, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:359848}, end_lsn:{lsn:359970}, log_id:2952, min_scn:{val:1713774197320369684, v:0}, max_scn:{val:1713774197320369684, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359726}, prev_proposal_id:8, committed_end_lsn:{lsn:359848}, data_checksum:667309872, accum_checksum:4099157314, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303053, freeze_ts:1713774194303053, submit_ts:1713774194303056, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.306327] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=90] no need update log(log_id=2952, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:66, proposal_id:8, committed_lsn:{lsn:359848}, max_scn:{val:1713774197320369684, v:0}, accumulated_checksum:4099157314, log_id:2952, flag:0}, log_task={header:{begin_lsn:{lsn:359848}, end_lsn:{lsn:359970}, log_id:2952, min_scn:{val:1713774197320369684, v:0}, max_scn:{val:1713774197320369684, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:359726}, prev_proposal_id:8, committed_end_lsn:{lsn:359848}, data_checksum:667309872, accum_checksum:4099157314, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303053, freeze_ts:1713774194303053, submit_ts:1713774194303056, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.306395] INFO [PALF] need_update_log_task_ (log_sliding_window.cpp:3608) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=60] receive log with same proposal_id(bool_ret=false, palf_id=1, self="172.23.88.129:2882", log_id=2953, log_end_lsn={lsn:360083}, need_send_ack=false, old_pid=8, log_task={header:{begin_lsn:{lsn:359970}, end_lsn:{lsn:360083}, log_id:2953, min_scn:{val:1713774197406167741, v:0}, max_scn:{val:1713774197406167741, v:0}, data_len:57, proposal_id:8, prev_lsn:{lsn:359848}, prev_proposal_id:8, committed_end_lsn:{lsn:359970}, data_checksum:2031283194, accum_checksum:295676114, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303063, freeze_ts:1713774194303063, submit_ts:1713774194303066, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}, header={magic:18258, version:1, group_size:57, proposal_id:8, committed_lsn:{lsn:359970}, max_scn:{val:1713774197406167741, v:0}, accumulated_checksum:295676114, log_id:2953, flag:0}, max_flushed_end_lsn={lsn:358576}, log_task={header:{begin_lsn:{lsn:359970}, end_lsn:{lsn:360083}, log_id:2953, min_scn:{val:1713774197406167741, v:0}, max_scn:{val:1713774197406167741, v:0}, data_len:57, proposal_id:8, prev_lsn:{lsn:359848}, prev_proposal_id:8, committed_end_lsn:{lsn:359970}, data_checksum:2031283194, accum_checksum:295676114, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303063, freeze_ts:1713774194303063, submit_ts:1713774194303066, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.306520] INFO [PALF] receive_log (log_sliding_window.cpp:3233) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1B-0-0] [lt=123] no need update log(log_id=2953, palf_id=1, self="172.23.88.129:2882", need_send_ack=false, is_log_pid_match=true, is_local_log_valid=true, is_prev_log_exist=true, is_prev_log_match=true, group_entry_header={magic:18258, version:1, group_size:57, proposal_id:8, committed_lsn:{lsn:359970}, max_scn:{val:1713774197406167741, v:0}, accumulated_checksum:295676114, log_id:2953, flag:0}, log_task={header:{begin_lsn:{lsn:359970}, end_lsn:{lsn:360083}, log_id:2953, min_scn:{val:1713774197406167741, v:0}, max_scn:{val:1713774197406167741, v:0}, data_len:57, proposal_id:8, prev_lsn:{lsn:359848}, prev_proposal_id:8, committed_end_lsn:{lsn:359970}, data_checksum:2031283194, accum_checksum:295676114, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1713774194303063, freeze_ts:1713774194303063, submit_ts:1713774194303066, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:0}) [2024-04-22 16:23:14.306778] INFO [PALF] after_flush_mode_meta (log_mode_mgr.cpp:771) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1C-0-0] [lt=52] after_flush_mode_meta success(ret=0, palf_id=1, self="172.23.88.129:2882", is_applied_mode_meta=true, mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, applied_mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:14.306842] INFO [PALF] receive_mode_meta (palf_handle_impl.cpp:2793) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1C-0-0] [lt=61] receive_mode_meta finish(ret=0, ret="OB_SUCCESS", this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, server="172.23.88.130:2882", proposal_id=8, is_applied_mode_meta=true, mode_meta={version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}) [2024-04-22 16:23:14.306911] INFO [PALF] handle_request (log_request_handler.cpp:335) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0A1C2F1C-0-0] [lt=66] receive_mode_meta success(ret=0, palf_id=1, server="172.23.88.130:2882", req={msg_proposal_id:8, meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, is_applied_mode_meta:true}, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:12, flying_meta_task:0}}) [2024-04-22 16:23:14.307890] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=141] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=5260, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:14.307953] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=168) [2024-04-22 16:23:14.308012] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=90, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:14.308066] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=49] [PALF STAT LOG IO WORKER QUEUE SIZE](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=11, this=0x7f3ce8f43380, extra_info="log_io_worker_num:1, cb_thread_pool_tg_id:296") [2024-04-22 16:23:14.308308] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=26, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:14.308367] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT LOG LIFE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=5851, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:14.308395] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=1, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=102, avg_log_submit_to_flush_time=5629, avg_log_submit_to_slide_time=5749) [2024-04-22 16:23:14.308376] INFO [PALF] read (log_cache.cpp:100) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=59] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4402, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:14.308428] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=270, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:14.308657] INFO [LIB] stat (utility.h:1165) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=29] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=124, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:14.308677] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=183] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=3, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:14.308812] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:882) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=83] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:4, post_barrier_lsn_:{lsn:358632}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774194308285, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:358576}, next_to_submit_scn_:{val:1713774177880657608, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:56, curr_read_buf_start_pos:0, curr_read_buf_end_pos:168, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358744}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:112, prev_entry_scn:{val:1713774187421454784, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:80, scn_:{val:1713774187421454784, v:0}, data_checksum:1754599746, flag:0}}, init_mode_version:0, accumulate_checksum:3241726231, curr_entry_is_padding:0, padding_entry_size:112, padding_entry_scn:{val:1713774187421454784, v:0}}}}}, replay_task={ls_id_:{id:1}, log_type:1, log_type:"TRANS_SERVICE", lsn_:{lsn:358632}, scn_:{val:1713774187421454784, v:0}, is_pre_barrier_:true, is_post_barrier_:true, log_size_:80, replay_hint_:7435878334736746667, is_raw_write_:false, first_handle_ts_:1713774194308808, replay_cost_:-1, retry_cost_:-1, log_buf_:0x7f3cbfc5f1a0}) [2024-04-22 16:23:14.309501] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1127) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=50] submit pre barrier log success(log_type=1, ls_id={id:1}, lsn={lsn:358632}, scn={val:1713774187421454784, v:0}, is_pre_barrier=true, is_post_barrier=true, log_size=80) [2024-04-22 16:23:14.309621] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1172) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=110] succ to submit log task to replay service(to_submit_lsn={lsn:358632}, to_submit_scn={val:1713774187421454784, v:0}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:3, post_barrier_lsn_:{lsn:358632}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774194308285, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:358744}, next_to_submit_scn_:{val:1713774187421454785, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:168, curr_read_buf_start_pos:0, curr_read_buf_end_pos:168, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358744}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187421454784, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:80, scn_:{val:1713774187421454784, v:0}, data_checksum:1754599746, flag:0}}, init_mode_version:0, accumulate_checksum:3241726231, curr_entry_is_padding:0, padding_entry_size:112, padding_entry_scn:{val:1713774187421454784, v:0}}}}}) [2024-04-22 16:23:14.311311] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:10, local_retry_times:10, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.311408] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=76][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.311469] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=58][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.311490] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.311523] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.312534] INFO [STORAGE.TRANS] replay_start_working_log (ob_trans_ctx_mgr_v4.cpp:676) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=176] [LsTxCtxMgr Role Change] replay start working log success(tenant_id_=1, ls_id_={id:1}) [2024-04-22 16:23:14.312630] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:896) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=74] pre barrier log replay succ(replay_task={ls_id_:{id:1}, log_type:1, log_type:"TRANS_SERVICE", lsn_:{lsn:358632}, scn_:{val:1713774187421454784, v:0}, is_pre_barrier_:true, is_post_barrier_:true, log_size_:80, replay_hint_:7435878334736746667, is_raw_write_:false, first_handle_ts_:1713774194309521, replay_cost_:3103, retry_cost_:-1, log_buf_:0x7f3cbfc5f1b0}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:358632}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774194308285, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:358744}, next_to_submit_scn_:{val:1713774187421454785, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:168, curr_read_buf_start_pos:0, curr_read_buf_end_pos:168, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358744}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187421454784, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:80, scn_:{val:1713774187421454784, v:0}, data_checksum:1754599746, flag:0}}, init_mode_version:0, accumulate_checksum:3241726231, curr_entry_is_padding:0, padding_entry_size:112, padding_entry_scn:{val:1713774187421454784, v:0}}}}}, replay_queue_idx=11, ret=0) [2024-04-22 16:23:14.312782] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:911) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=148] post barrier log replay succ(replay_task={ls_id_:{id:1}, log_type:1, log_type:"TRANS_SERVICE", lsn_:{lsn:358632}, scn_:{val:1713774187421454784, v:0}, is_pre_barrier_:true, is_post_barrier_:true, log_size_:80, replay_hint_:7435878334736746667, is_raw_write_:false, first_handle_ts_:1713774194309521, replay_cost_:3103, retry_cost_:-1, log_buf_:null}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774194308285, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:358744}, next_to_submit_scn_:{val:1713774187421454785, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:168, curr_read_buf_start_pos:0, curr_read_buf_end_pos:168, log_storage_:{IteratorStorage:{start_lsn:{lsn:358576}, end_lsn:{lsn:358744}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774187421454784, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:80, scn_:{val:1713774187421454784, v:0}, data_checksum:1754599746, flag:0}}, init_mode_version:0, accumulate_checksum:3241726231, curr_entry_is_padding:0, padding_entry_size:112, padding_entry_scn:{val:1713774187421454784, v:0}}}}}, ret=0) [2024-04-22 16:23:14.314094] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:721) [10869][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=24] update id meta(ret=0, service_type=0, limited_id=1713774207210958129, latest_scn={val:1713774197210958129, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774207210958129, latest_log_ts:{val:1713774197210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}) [2024-04-22 16:23:14.314286] INFO [STORAGE.TRANS] handle_replay_result (ob_id_service.cpp:237) [10869][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=187] handle reply result(ret=0, service_type_=0, last_id=1713774207210958129, limited_id=1713774207210958129, log_ts={val:1713774197210958129, v:0}) [2024-04-22 16:23:14.314331] INFO [STORAGE.TRANS] replay (ob_id_service.cpp:210) [10869][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=41] ObIDService replay(service_type_=0, ret=0, ls_log={last_id_:1713774207210958129, limit_id_:1713774207210958129}, base_header={version:1, log_type:7, flag:0, replay_hint:0}) [2024-04-22 16:23:14.315549] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] [2024-04-22 16:23:14.315590] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.315611] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.315628] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.315639] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.315647] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.315654] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.315692] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.315733] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=35][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.315748] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.315779] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=11, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.315805] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13] will sleep(sleep_us=11000, remain_us=1892605, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.326983] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:11, local_retry_times:11, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.327064] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.327089] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.327106] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.327135] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.380292] INFO keepalive_check (time_wheel.c:76) [10460][pnio1][T0][Y0-0000000000000000-0-0] [lt=31] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd332c048,dest=-2124933204:2882), old=0, new=3000000 [2024-04-22 16:23:14.380284] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:61, local_retry_times:61, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.380310] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:111) [10528][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4015] clock out of order(cur_ts=1713774194380307, cur_ts_=1713774194334024, delta=46283) [2024-04-22 16:23:14.380328] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.380334] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.380338] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.380348] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=4][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.380398] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=24][errcode=-4725] [2024-04-22 16:23:14.380420] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.380446] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.380454] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.380463] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.380469] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.380475] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.380504] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.380525] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.380534] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.380544] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=12, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.380561] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8] will sleep(sleep_us=12000, remain_us=1827848, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.381684] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] [2024-04-22 16:23:14.381698] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.381706] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.381712] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.381717] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.381720] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.381729] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.381763] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=3] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.381781] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.381787] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:14.381793] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=62, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.381805] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6] will sleep(sleep_us=62000, remain_us=20445337, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=62, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.392658] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:12, local_retry_times:12, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.392692] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.392700] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.392704] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=4][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.392745] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=4][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.393934] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] [2024-04-22 16:23:14.393947] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.393959] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.393965] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.393971] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.393975] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=4][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.393980] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.394013] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.394033] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.394042] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.394053] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=13, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.394068] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] will sleep(sleep_us=13000, remain_us=1814340, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.394074] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFC17-0-0] [lt=83][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774194393997}) [2024-04-22 16:23:14.400059] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:14.400083] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:14.400123] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774192906167000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1713774194400052) [2024-04-22 16:23:14.400152] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774194200021, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:14.400197] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=-2920758, log_service_ts=1713774197320369, min_tx_service_ts=4611686018427387, timestamp={val:1713774197320369684, v:0}) [2024-04-22 16:23:14.400221] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774187313945245, v:0}", last_lsn={lsn:358397}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:14.407173] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:13, local_retry_times:13, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.407219] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.407233] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.407241] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.407260] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.409676] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=71] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:14.410819] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] Cache replace map node details(ret=0, replace_node_count=0, replace_time=6207, replace_start_pos=880796, replace_num=62914) [2024-04-22 16:23:14.410856] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:14.411099] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] [2024-04-22 16:23:14.411138] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.411162] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.411176] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.411188] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.411197] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.411206] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.411247] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.411283] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.411300] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.411321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=14, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.411349] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=14000, remain_us=1797060, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.425506] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:14, local_retry_times:14, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.425586] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=58][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.425634] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.425651] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.425709] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.429552] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=76][errcode=-4725] [2024-04-22 16:23:14.429605] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=50][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.429647] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.429671] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.429691] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.429708] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.429723] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.429861] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=80] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.429930] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=60][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.429961] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.429991] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=15, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.430036] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28] will sleep(sleep_us=15000, remain_us=1778374, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.443939] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:62, local_retry_times:62, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.444053] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=85][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.444081] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.444101] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.444144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.445284] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:15, local_retry_times:15, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.445372] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.445408] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.445428] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.445517] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=73][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.448843] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] [2024-04-22 16:23:14.448890] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.448922] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.448940] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.448955] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.448966] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.448977] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.449026] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.449080] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.449117] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:14.449135] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=63, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.449170] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=63000, remain_us=20377973, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=63, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.450241] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=77][errcode=-4725] [2024-04-22 16:23:14.450291] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.450323] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.450345] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.450367] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.450383] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.450399] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.450526] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.450609] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=73][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.450633] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.450654] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=16, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.450687] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=16000, remain_us=1757722, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.466901] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=61][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:16, local_retry_times:16, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.466992] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.467030] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.467051] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.467115] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.471631] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=50][errcode=-4725] [2024-04-22 16:23:14.471686] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=52][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.471715] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.471729] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.471743] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.471754] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.471785] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.471835] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.471911] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=68][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.471933] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.471959] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=17, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.472007] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=31] will sleep(sleep_us=17000, remain_us=1736403, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.489204] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=35][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:17, local_retry_times:17, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.489294] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.489333] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.489352] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.489386] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.493899] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=120][errcode=-4725] [2024-04-22 16:23:14.493947] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.493979] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.494003] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.494019] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.494030] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.494041] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.494101] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.494157] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.494179] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:14.494227] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=18, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:14.494321] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=74] will sleep(sleep_us=18000, remain_us=1714090, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.500192] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:14.500261] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197320369684, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774194500179) [2024-04-22 16:23:14.500298] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774194400165, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:14.512314] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=37][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:63, local_retry_times:63, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.512400] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.512428] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.512479] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=49][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.512506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.512515] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=43][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:18, local_retry_times:18, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:14.512577] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=43][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.512618] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:14.512638] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:14.512666] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:14.516613] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=24][errcode=-4725] [2024-04-22 16:23:14.516667] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=51][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.516699] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.516717] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.516732] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.516733] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:14.516743] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.516776] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.516756] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:14.516822] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:14.516838] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:14.516852] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:14.516864] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:14.516848] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.516875] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.516887] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:14.516921] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=1691489, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.516950] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.517012] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=64000, remain_us=20310130, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=64, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.540545] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.540632] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1667777, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.565599] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.565688] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=21000, remain_us=1642721, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.585725] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.585857] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=65000, remain_us=20241286, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=65, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.590763] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.590855] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=22000, remain_us=1617555, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.610319] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=44] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:14.610987] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=16] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:14.617264] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.617358] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=23000, remain_us=1591051, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.645254] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.645348] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1563061, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.653921] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.654008] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=66000, remain_us=20173134, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=66, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.673320] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.673405] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=1535004, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.675658] INFO [SERVER] fetch_sys_ls (ob_service.cpp:663) [10951][T1_L0_G0][T1][YB42AC175882-000616AB0B3C2F3B-0-0] [lt=3] fetch sys_ls succeed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2024-04-22 16:23:14.681300] INFO keepalive_check (time_wheel.c:76) [10461][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3d01bf8048,dest=-2091378772:2882), old=0, new=3000000 [2024-04-22 16:23:14.700388] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:14.700481] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=89][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197320369684, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774194700375) [2024-04-22 16:23:14.700518] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774194500321, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:14.700559] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:14.702316] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.702401] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=26000, remain_us=1506009, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.704170] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=98] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:14.724305] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.724390] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=67000, remain_us=20102752, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=67, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.732530] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.732612] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=27000, remain_us=1475797, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.734546] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3588-0-0] [lt=191][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:14.734611] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3588-0-0] [lt=61][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:14.734681] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3588-0-0] [lt=66][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:14.734774] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3588-0-0] [lt=90][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:14.764913] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.764995] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=28000, remain_us=1443415, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.796282] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.796343] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=29000, remain_us=1412066, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.797348] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.797503] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=68000, remain_us=20029640, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=68, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.811091] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=103] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:14.811134] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:14.827140] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.827177] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=30000, remain_us=1381232, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.861088] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.861178] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=31000, remain_us=1347231, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.869883] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.869989] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=69000, remain_us=19957153, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=69, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.894688] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.894729] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=32000, remain_us=1313679, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.900613] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:14.900711] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197320369684, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774194900601) [2024-04-22 16:23:14.900752] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774194700548, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:14.930208] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.930242] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=33000, remain_us=1278167, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.939144] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:14.945272] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.945389] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=70000, remain_us=19881753, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=70, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:14.967607] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:14.967683] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1240726, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:14.972989] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10605][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=75][errcode=0] server is initiating(server_id=0, local_seq=14852, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:14.973095] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10605][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=102] new mysql sessid created(conn.sessid_=3221240324, support_ssl=false) [2024-04-22 16:23:14.973170] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10605][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221240324, sess.client_addr_="172.23.88.134:41994") [2024-04-22 16:23:14.973206] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10605][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=27] accept one succ(*s={this:0x7f3cf8b72970, session_id:3221240324, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:237}, err:0, last_decode_time:0, last_write_time:1713774194973141, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:14.973595] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10605][sql_nio4][T1][Y0-0000000000000000-0-0] [lt=69] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:14.973621] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [10605][sql_nio4][T1][Y0-0000000000000000-0-0] [lt=25] mysql login queue(mysql_queue->queue_.size()=1) [2024-04-22 16:23:14.973712] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=9] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:14.974487] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=40] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:14.974527] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=32] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:14.974540] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=11][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"'PA2v|-gCUcp-,6h,ABg"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:23:14.974560] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=18][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"'PA2v|-gCUcp-,6h,ABg"}, ret=-4043) [2024-04-22 16:23:14.974575] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=12][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:23:14.974587] WDIAG [SERVER] process (obmp_connect.cpp:346) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=10][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:23:14.974645] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=8] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:14.974758] INFO [SERVER] free_session (obmp_base.cpp:326) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=21] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221240324, proxy_sessid:0}) [2024-04-22 16:23:14.974790] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=23][errcode=0] server close connection(sessid=3221240324, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:14.974806] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221240324, proxy_sessid=0) [2024-04-22 16:23:14.974817] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=8][errcode=-4016] session is null [2024-04-22 16:23:14.974828] INFO [SERVER] process (obmp_connect.cpp:488) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.23.88.134", client_ip=172.23.88.134, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221240324, proxy_sessid=0, client_sessid=3221240324, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=true, capability=683647754, proxy_capability=1480, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:23:14.974875] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=43][errcode=-4018] get session fail(ret=-4018, sessid=3221240324, proxy_sessid=0) [2024-04-22 16:23:14.974887] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=10][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:14.974921] WDIAG [SERVER] response (obmp_base.cpp:89) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=8][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:14.974941] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDC-0-0] [lt=18][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:14.974968] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10605][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f3cf8b72970, session_id:3221240324, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:237}, err:5, last_decode_time:1713774194973512, last_write_time:1713774194974966, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:378, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:14.975024] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10605][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=39] kill and revert session(conn.sessid_=3221240324, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:14.975039] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10605][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f3cf8b72970, session_id:3221240324, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:237}, err:5, last_decode_time:1713774194973512, last_write_time:1713774194974966, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:378, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:14.975067] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10605][sql_nio4][T0][Y0-0000000000000000-0-0] [lt=24] connection close(sessid=3221240324, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:14.977491] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10601][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] server is initiating(server_id=0, local_seq=14853, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:14.977558] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10601][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=63] new mysql sessid created(conn.sessid_=3221240325, support_ssl=false) [2024-04-22 16:23:14.977667] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10601][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221240325, sess.client_addr_="172.23.88.134:41996") [2024-04-22 16:23:14.977760] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10601][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=63] accept one succ(*s={this:0x7f3ce8bfc030, session_id:3221240325, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:243}, err:0, last_decode_time:0, last_write_time:1713774194977614, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:14.978121] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10601][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=122] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:14.978228] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDE-0-0] [lt=8] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:14.979232] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDE-0-0] [lt=41] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:14.979278] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDE-0-0] [lt=38] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:14.979293] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDE-0-0] [lt=13][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"!3|client_type_=3, conn->client_version_=0) [2024-04-22 16:23:14.979781] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDE-0-0] [lt=45][errcode=-4018] get session fail(ret=-4018, sessid=3221240325, proxy_sessid=0) [2024-04-22 16:23:14.979799] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDE-0-0] [lt=16][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:14.979841] WDIAG [SERVER] response (obmp_base.cpp:89) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDE-0-0] [lt=21][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:14.979866] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10833][T1_MysqlQueueTh][T1][Y0-000616AB08CFEBDE-0-0] [lt=23][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:14.979884] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10601][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4015] close sql sock by user req(*s={this:0x7f3ce8bfc030, session_id:3221240325, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:243}, err:5, last_decode_time:1713774194978056, last_write_time:1713774194979882, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:14.979925] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10601][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=34] kill and revert session(conn.sessid_=3221240325, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:14.979947] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10601][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] can close safely, do destroy(*s={this:0x7f3ce8bfc030, session_id:3221240325, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:243}, err:5, last_decode_time:1713774194978056, last_write_time:1713774194979882, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:14.979964] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10601][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221240325, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:14.991638] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] server is initiating(server_id=0, local_seq=14854, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:14.991723] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=80] new mysql sessid created(conn.sessid_=3221240326, support_ssl=false) [2024-04-22 16:23:14.991803] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221240326, sess.client_addr_="172.23.88.134:42014") [2024-04-22 16:23:14.991849] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=33] accept one succ(*s={this:0x7f3cf8b6c030, session_id:3221240326, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:244}, err:0, last_decode_time:0, last_write_time:1713774194991779, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:14.992238] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10603][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=83] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:14.992403] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=8] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:14.993855] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=100] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:14.993914] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=48] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:14.993953] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=35][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"](K$n#\(RD>I7sakw5>y"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:23:14.994003] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=45][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"](K$n#\(RD>I7sakw5>y"}, ret=-4043) [2024-04-22 16:23:14.994059] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=51][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:23:14.994095] WDIAG [SERVER] process (obmp_connect.cpp:346) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=32][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:23:14.994151] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=15] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:14.994274] INFO [SERVER] free_session (obmp_base.cpp:326) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=35] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221240326, proxy_sessid:0}) [2024-04-22 16:23:14.994327] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=39][errcode=0] server close connection(sessid=3221240326, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f3d1c738ea5 0x7f3d1c461b0d") [2024-04-22 16:23:14.994363] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=32][errcode=-4018] get session fail(ret=-4018, sessid=3221240326, proxy_sessid=0) [2024-04-22 16:23:14.994396] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=30][errcode=-4016] session is null [2024-04-22 16:23:14.994419] INFO [SERVER] process (obmp_connect.cpp:488) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=18] MySQL LOGIN(direct_client_ip="172.23.88.134", client_ip=172.23.88.134, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221240326, proxy_sessid=0, client_sessid=3221240326, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=true, capability=683647754, proxy_capability=1480, use_ssl=false, c/s protocol="OB_2_0_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:23:14.994550] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=125][errcode=-4018] get session fail(ret=-4018, sessid=3221240326, proxy_sessid=0) [2024-04-22 16:23:14.994591] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=38][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:14.994649] WDIAG [SERVER] response (obmp_base.cpp:89) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=29][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:14.994680] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10831][T1_MysqlQueueTh][T1][Y0-000616AB08BFEBDA-0-0] [lt=28][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:14.994725] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4015] close sql sock by user req(*s={this:0x7f3cf8b6c030, session_id:3221240326, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:244}, err:5, last_decode_time:1713774194992118, last_write_time:1713774194994721, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:378, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:14.994800] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=62] kill and revert session(conn.sessid_=3221240326, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:14.994839] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=36] can close safely, do destroy(*s={this:0x7f3cf8b6c030, session_id:3221240326, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:244}, err:5, last_decode_time:1713774194992118, last_write_time:1713774194994721, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:378, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:14.994905] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10603][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=61] connection close(sessid=3221240326, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:14.999129] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] server is initiating(server_id=0, local_seq=14855, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:23:14.999220] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=86] new mysql sessid created(conn.sessid_=3221240327, support_ssl=false) [2024-04-22 16:23:14.999323] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221240327, sess.client_addr_="172.23.88.134:42016") [2024-04-22 16:23:14.999371] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=34] accept one succ(*s={this:0x7f3cf8b6cd70, session_id:3221240327, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:245}, err:0, last_decode_time:0, last_write_time:1713774194999300, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:14.999756] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [10607][sql_nio6][T1][Y0-0000000000000000-0-0] [lt=38] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:23:14.999868] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDD-0-0] [lt=10] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:23:15.001007] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDD-0-0] [lt=60] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:23:15.001059] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDD-0-0] [lt=43] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:23:15.001081] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDD-0-0] [lt=20][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"172.23.88.134", db:"oceanbase", scramble_str:"nx2/2Ow)b5~=VvoJe<client_type_=3, conn->client_version_=0) [2024-04-22 16:23:15.001415] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDD-0-0] [lt=45][errcode=-4018] get session fail(ret=-4018, sessid=3221240327, proxy_sessid=0) [2024-04-22 16:23:15.001461] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDD-0-0] [lt=18][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:23:15.001500] WDIAG [SERVER] response (obmp_base.cpp:89) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDD-0-0] [lt=13][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:23:15.001513] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [10832][T1_MysqlQueueTh][T1][Y0-000616AB089FEBDD-0-0] [lt=12][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:23:15.001598] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4015] close sql sock by user req(*s={this:0x7f3cf8b6cd70, session_id:3221240327, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:245}, err:5, last_decode_time:1713774194999607, last_write_time:1713774195001594, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:23:15.001702] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=88] kill and revert session(conn.sessid_=3221240327, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:23:15.001761] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=30] can close safely, do destroy(*s={this:0x7f3cf8b6cd70, session_id:3221240327, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:245}, err:5, last_decode_time:1713774194999607, last_write_time:1713774195001594, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:381, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:23:15.001909] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [10607][sql_nio6][T0][Y0-0000000000000000-0-0] [lt=142] connection close(sessid=3221240327, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_2_0_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:23:15.004070] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.004158] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1204252, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.005470] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [10532][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=47] runTimer to refresh locality_info(ret=0) [2024-04-22 16:23:15.007862] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.007851] INFO [CLOG] operator() (ob_log_replay_service.cpp:1492) [10817][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=19] get_replay_process success(id={id:1}, replayed_log_size=359970, unreplayed_log_size=0) [2024-04-22 16:23:15.008027] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:156) [10817][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=106] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-04-22 16:23:15.008018] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=98] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:15.008105] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=80][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:15.008131] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=23][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:15.008221] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=86][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:15.008312] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=82][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:15.008327] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:15.008345] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:23:15.008382] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=33][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:15.008421] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=35][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:15.008511] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=87][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:15.008542] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=28][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:23:15.008570] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=26][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:23:15.008598] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=25][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:23:15.008625] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=24][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:15.008654] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=26][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:15.008683] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=25][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:15.008711] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=24][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:15.008756] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=29][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:15.008802] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=38][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.008847] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=39][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.008882] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=32][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:15.008911] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=25][errcode=-5019] fail to handle text query(stmt=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name, ret=-5019) [2024-04-22 16:23:15.008957] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=42][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:15.008986] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=26][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.009030] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=37][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:15.009078] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=39][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.009098] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.009125] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=25][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:15.009179] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=27][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:15.009216] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10531][LocltyRefTask][T0][YB42AC175881-000616AB0AEFEBDE-0-0] [lt=32][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, aret=-5019, ret=-5019) [2024-04-22 16:23:15.009258] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-04-22 16:23:15.009290] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:15.009322] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:15.009360] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-5019] query failed(ret=-5019, conn=0x7f3ceeb84050, start=1713774195007280, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-04-22 16:23:15.009412] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:15.009484] WDIAG [SHARE] load_region (ob_locality_table_operator.cpp:158) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-5019] execute sql failed(ret=-5019, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-04-22 16:23:15.009527] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=40] load region(ret=-5019, locality_info={version:0, local_region:"", local_zone:"", local_idc:"", local_zone_type:3, local_zone_status:3, locality_region_array:[], locality_zone_array:[]}) [2024-04-22 16:23:15.009684] WDIAG [STORAGE] load_region (ob_locality_manager.cpp:236) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-5019] localitity operator load region error(ret=-5019) [2024-04-22 16:23:15.009721] WDIAG [STORAGE] process (ob_locality_manager.cpp:762) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-5019] process refresh locality task fail(ret=-5019) [2024-04-22 16:23:15.009958] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [10521][DumpLSLoc][T0][YB42AC175881-000616AB0AFFEBDE-0-0] [lt=85] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774193200889, replica_locations:[{server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:8}]}]) [2024-04-22 16:23:15.011413] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:15.011784] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=88] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:15.017905] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.017970] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=71000, remain_us=19809172, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=71, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.043076] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.043172] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=36000, remain_us=1165237, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.064194] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=13][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.064274] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=59] table not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, table_name.ptr()="data_size:25, data:5F5F616C6C5F6469736B5F696F5F63616C6962726174696F6E", ret=-5019) [2024-04-22 16:23:15.064315] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=35][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, ret=-5019) [2024-04-22 16:23:15.064343] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=25][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_disk_io_calibration, db_name=oceanbase) [2024-04-22 16:23:15.064370] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=25][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_disk_io_calibration) [2024-04-22 16:23:15.064393] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:15.064406] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:15.064423] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_disk_io_calibration' doesn't exist [2024-04-22 16:23:15.064474] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=48][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:15.064488] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:15.064499] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:15.064521] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=20][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:15.064534] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:15.064546] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:15.064558] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:15.064583] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:15.064597] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.064612] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.064624] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:15.064636] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA", ret=-5019) [2024-04-22 16:23:15.064651] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:15.064664] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.064709] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=38][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:15.064765] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=47][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.064780] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.064790] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:15.064813] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC43-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA""}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:15.064843] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10404][ServerGTimer][T0][YB42AC175881-000616AB07DFEC43-0-0] [lt=25][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA""}, aret=-5019, ret=-5019) [2024-04-22 16:23:15.064862] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:23:15.064877] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:15.064890] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:15.064903] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45f0050, start=1713774195063857, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:23:15.064921] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:15.064944] WDIAG [COMMON] parse_calibration_table (ob_io_calibration.cpp:838) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] query failed(ret=-5019, sql_string=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:23:15.065043] WDIAG [COMMON] read_from_table (ob_io_calibration.cpp:708) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] parse calibration data failed(ret=-5019) [2024-04-22 16:23:15.065077] WDIAG [SERVER] refresh_io_calibration (ob_server.cpp:3426) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-5019] fail to refresh io calibration from table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.065099] WDIAG [SERVER] runTimerTask (ob_server.cpp:3416) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] ObRefreshIOCalibrationTimeTask task failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.082317] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.082390] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=37000, remain_us=1126020, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.092779] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.092869] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=72000, remain_us=19734273, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=72, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.100858] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:15.100928] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197320369684, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774195100845) [2024-04-22 16:23:15.100960] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774194900772, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:15.112042] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=32][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.112161] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=94] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-04-22 16:23:15.112194] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-04-22 16:23:15.112214] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-04-22 16:23:15.112244] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=28][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-04-22 16:23:15.112263] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:15.112276] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:15.112315] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=17][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-04-22 16:23:15.112357] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=38][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:15.112381] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=22][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:15.112392] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:15.112404] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:15.112415] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:15.112427] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:15.112484] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=54][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:15.112531] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=32][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:15.112574] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=38][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.112602] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=23][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.112617] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=13][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:15.112636] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=16][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-04-22 16:23:15.112659] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=20][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:15.112678] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=16][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.112733] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=47][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:15.112770] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=29][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.112787] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=15][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.112802] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:15.112862] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=24][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:15.112909] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=41][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:15.112933] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=21][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:23:15.112951] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=16][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:15.112971] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=18][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:15.112991] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=18][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1498050, start=1713774195111147, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:23:15.113028] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=33][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:15.113047] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1177) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=14][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-04-22 16:23:15.113178] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:872) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=19][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:23:15.113228] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:68) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=43][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.113261] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=28][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.113308] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:94) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=43][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.113346] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:108) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=35] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.114824] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=48][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.114873] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=37] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-04-22 16:23:15.114894] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=19][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-04-22 16:23:15.114907] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-04-22 16:23:15.114921] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-04-22 16:23:15.114933] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:15.114942] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE1-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:15.115214] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:301) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=0] cal_all_part_disk_default_percentage succ(data_dir="/home/creality/oceanbase/store/sstable", clog_dir="/home/creality/oceanbase/store/clog", shared_mode=false, data_disk_total_size=205110317056, data_disk_default_percentage=90, clog_disk_total_size=85856358400, clog_disk_default_percentage=90) [2024-04-22 16:23:15.115252] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:337) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=36] decide disk size finished(suggested_disk_size=81604378624, suggested_disk_percentage=0, default_disk_percentage=90, total_space=85856358400, disk_size=81604378624) [2024-04-22 16:23:15.115281] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=27] get_log_disk_info_in_config(suggested_data_disk_size=161061273600, suggested_clog_disk_size=81604378624, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=81604378624, log_disk_percentage=0, total_log_disk_size=85856358400) [2024-04-22 16:23:15.115299] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=14] try_resize success(ret=0, log_disk_size=81604378624, total_log_disk_size=85856358400, this={dir::"/home/creality/oceanbase/store/clog/log_pool", dir_fd:14, meta_fd:15, log_pool_meta:{curr_total_size:81604378624, next_total_size:81604378624, status:0}, min_block_id:2, max_block_id:1216, min_log_disk_size_for_all_tenants_:4294967296, is_inited:true}) [2024-04-22 16:23:15.115326] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:132) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE1-0-0] [lt=25] refresh tenant config(tenants=[], ret=-5019) [2024-04-22 16:23:15.122287] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.122515] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=38000, remain_us=1085896, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.131882] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=67] === Run print tenant memory usage task === [2024-04-22 16:23:15.131943] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=26] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 579,035,136 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 192,438,272 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 115,343,360 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 12,582,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 33,783,808 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 201,011,200 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:15.132261] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 192,438,272 used= 181,620,464 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,477,760 used= 31,458,304 count= 1 avg_used= 31,458,304 block_cnt= 1 chunk_cnt= 1 mod=ash_list [MEMORY] hold= 22,761,472 used= 22,699,200 count= 9 avg_used= 2,522,133 block_cnt= 9 chunk_cnt= 8 mod=IoControl [MEMORY] hold= 16,883,712 used= 15,771,688 count= 229 avg_used= 68,872 block_cnt= 229 chunk_cnt= 31 mod=[T]ObSessionDIB [MEMORY] hold= 12,873,728 used= 12,856,352 count= 1 avg_used= 12,856,352 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 9,457,664 used= 9,437,784 count= 1 avg_used= 9,437,784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckInteChe [MEMORY] hold= 5,513,216 used= 5,502,336 count= 21 avg_used= 262,016 block_cnt= 21 chunk_cnt= 9 mod=ObTxDesc [MEMORY] hold= 5,331,904 used= 5,240,872 count= 49 avg_used= 106,956 block_cnt= 25 chunk_cnt= 11 mod=PartitTableTask [MEMORY] hold= 5,267,456 used= 5,190,496 count= 8 avg_used= 648,812 block_cnt= 8 chunk_cnt= 3 mod=KvstCachWashStr [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashPxBlooFilKe [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=PxP2PDhMgrKey [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBucTenComMo [MEMORY] hold= 4,307,328 used= 4,278,711 count= 3 avg_used= 1,426,237 block_cnt= 3 chunk_cnt= 3 mod=SqlDtlMgr [MEMORY] hold= 4,231,936 used= 4,203,008 count= 6 avg_used= 700,501 block_cnt= 5 chunk_cnt= 3 mod=BaseLogWriter [MEMORY] hold= 4,214,880 used= 4,194,328 count= 2 avg_used= 2,097,164 block_cnt= 2 chunk_cnt= 2 mod=SerFuncRegHT [MEMORY] hold= 3,276,800 used= 3,214,848 count= 11 avg_used= 292,258 block_cnt= 11 chunk_cnt= 7 mod=DedupQueue [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdUnitMa [MEMORY] hold= 2,167,040 used= 2,129,912 count= 7 avg_used= 304,273 block_cnt= 7 chunk_cnt= 5 mod=FixedQueue [MEMORY] hold= 2,114,080 used= 2,031,616 count= 1,026 avg_used= 1,980 block_cnt= 209 chunk_cnt= 3 mod=TabletLSMap [MEMORY] hold= 2,037,344 used= 1,915,008 count= 468 avg_used= 4,091 block_cnt= 243 chunk_cnt= 3 mod=CharsetInit [MEMORY] hold= 1,843,200 used= 1,638,400 count= 25 avg_used= 65,536 block_cnt= 25 chunk_cnt= 14 mod=[T]char [MEMORY] hold= 1,591,232 used= 1,560,960 count= 21 avg_used= 74,331 block_cnt= 20 chunk_cnt= 9 mod=TenantCtxAlloca [MEMORY] hold= 1,581,056 used= 1,572,904 count= 1 avg_used= 1,572,904 block_cnt= 1 chunk_cnt= 1 mod=IdConnMap [MEMORY] hold= 1,523,712 used= 1,507,118 count= 5 avg_used= 301,423 block_cnt= 5 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 1,452,608 used= 1,434,880 count= 44 avg_used= 32,610 block_cnt= 44 chunk_cnt= 20 mod=CommonArray [MEMORY] hold= 1,408,800 used= 1,388,000 count= 4 avg_used= 347,000 block_cnt= 4 chunk_cnt= 3 mod=Omt [MEMORY] hold= 1,286,144 used= 1,270,296 count= 2 avg_used= 635,148 block_cnt= 2 chunk_cnt= 2 mod=PxResMgr [MEMORY] hold= 1,140,864 used= 1,105,088 count= 557 avg_used= 1,984 block_cnt= 145 chunk_cnt= 24 mod=CreateContext [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=ConcurHashMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=TsSourceInfoMap [MEMORY] hold= 933,888 used= 897,024 count= 5 avg_used= 179,404 block_cnt= 5 chunk_cnt= 5 mod=LightyQueue [MEMORY] hold= 933,888 used= 917,576 count= 2 avg_used= 458,788 block_cnt= 2 chunk_cnt= 1 mod=CACHE_INST_MAP [MEMORY] hold= 827,472 used= 773,160 count= 13 avg_used= 59,473 block_cnt= 12 chunk_cnt= 8 mod=HashBucket [MEMORY] hold= 704,512 used= 691,200 count= 2 avg_used= 345,600 block_cnt= 2 chunk_cnt= 2 mod=LDBlockBitMap [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=EventTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=GEleTimer [MEMORY] hold= 622,592 used= 619,008 count= 1 avg_used= 619,008 block_cnt= 1 chunk_cnt= 1 mod=SysTaskStatus [MEMORY] hold= 614,400 used= 556,560 count= 15 avg_used= 37,104 block_cnt= 15 chunk_cnt= 8 mod=InneSqlConnPool [MEMORY] hold= 573,440 used= 304,920 count= 35 avg_used= 8,712 block_cnt= 35 chunk_cnt= 13 mod=[T]ObDSActionAr [MEMORY] hold= 512,000 used= 507,904 count= 64 avg_used= 7,936 block_cnt= 64 chunk_cnt= 2 mod=ModulePageAlloc [MEMORY] hold= 491,264 used= 441,808 count= 271 avg_used= 1,630 block_cnt= 116 chunk_cnt= 13 mod=tg [MEMORY] hold= 458,752 used= 457,856 count= 7 avg_used= 65,408 block_cnt= 7 chunk_cnt= 6 mod=ObTsResponseTas [MEMORY] hold= 448,000 used= 444,416 count= 56 avg_used= 7,936 block_cnt= 56 chunk_cnt= 13 mod=SessionInfoHash [MEMORY] hold= 425,984 used= 420,032 count= 1 avg_used= 420,032 block_cnt= 1 chunk_cnt= 1 mod=CompEventMgr [MEMORY] hold= 403,648 used= 378,880 count= 258 avg_used= 1,468 block_cnt= 54 chunk_cnt= 2 mod=LSLocationMap [MEMORY] hold= 401,408 used= 393,256 count= 1 avg_used= 393,256 block_cnt= 1 chunk_cnt= 1 mod=TablStorStatMgr [MEMORY] hold= 368,896 used= 330,944 count= 7 avg_used= 47,277 block_cnt= 7 chunk_cnt= 3 mod=Rpc [MEMORY] hold= 327,680 used= 325,960 count= 5 avg_used= 65,192 block_cnt= 5 chunk_cnt= 5 mod=LatchStat [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=register_tasks [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=register_task [MEMORY] hold= 262,144 used= 261,632 count= 4 avg_used= 65,408 block_cnt= 4 chunk_cnt= 4 mod=AsyncTaskQueue [MEMORY] hold= 229,376 used= 224,000 count= 1 avg_used= 224,000 block_cnt= 1 chunk_cnt= 1 mod=BGTMonitor [MEMORY] hold= 221,184 used= 212,992 count= 1 avg_used= 212,992 block_cnt= 1 chunk_cnt= 1 mod=TSWorker [MEMORY] hold= 221,184 used= 214,432 count= 1 avg_used= 214,432 block_cnt= 1 chunk_cnt= 1 mod=CompSuggestMgr [MEMORY] hold= 212,992 used= 196,624 count= 2 avg_used= 98,312 block_cnt= 2 chunk_cnt= 1 mod=DRTaskMap [MEMORY] hold= 212,992 used= 204,928 count= 1 avg_used= 204,928 block_cnt= 1 chunk_cnt= 1 mod=TenantMutilAllo [MEMORY] hold= 212,992 used= 196,624 count= 2 avg_used= 98,312 block_cnt= 2 chunk_cnt= 2 mod=DdlQue [MEMORY] hold= 203,872 used= 202,200 count= 26 avg_used= 7,776 block_cnt= 26 chunk_cnt= 6 mod=HashNode [MEMORY] hold= 180,224 used= 172,064 count= 1 avg_used= 172,064 block_cnt= 1 chunk_cnt= 1 mod=TenantMBList [MEMORY] hold= 163,840 used= 147,792 count= 2 avg_used= 73,896 block_cnt= 2 chunk_cnt= 1 mod=HashNodNexWaiMa [MEMORY] hold= 163,504 used= 155,416 count= 2 avg_used= 77,708 block_cnt= 2 chunk_cnt= 2 mod=OB_DISK_REP [MEMORY] hold= 160,624 used= 159,272 count= 21 avg_used= 7,584 block_cnt= 21 chunk_cnt= 2 mod=TenaSpaTabIdSet [MEMORY] hold= 155,648 used= 153,152 count= 3 avg_used= 51,050 block_cnt= 3 chunk_cnt= 2 mod=IO_MGR [MEMORY] hold= 155,648 used= 147,624 count= 1 avg_used= 147,624 block_cnt= 1 chunk_cnt= 1 mod=UsrRuleMap [MEMORY] hold= 144,880 used= 143,656 count= 19 avg_used= 7,560 block_cnt= 19 chunk_cnt= 2 mod=SysTableNameMap [MEMORY] hold= 139,264 used= 131,776 count= 1 avg_used= 131,776 block_cnt= 1 chunk_cnt= 1 mod=GtsTaskQueue [MEMORY] hold= 137,920 used= 134,912 count= 11 avg_used= 12,264 block_cnt= 11 chunk_cnt= 3 mod=SeArray [MEMORY] hold= 124,416 used= 122,688 count= 27 avg_used= 4,544 block_cnt= 27 chunk_cnt= 11 mod=[T]ObTraceEvent [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=TenaSchForCacMa [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=MemMgrForLiboMa [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=MemMgrMap [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=RefrFullScheMap [MEMORY] hold= 115,616 used= 114,656 count= 15 avg_used= 7,643 block_cnt= 15 chunk_cnt= 2 mod=HashNodeConfCon [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=NonPartTenMap [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=IndNameMap [MEMORY] hold= 114,336 used= 106,088 count= 2 avg_used= 53,044 block_cnt= 2 chunk_cnt= 2 mod=RetryCtrl [MEMORY] hold= 106,496 used= 98,312 count= 1 avg_used= 98,312 block_cnt= 1 chunk_cnt= 1 mod=TmpFileManager [MEMORY] hold= 106,496 used= 86,408 count= 5 avg_used= 17,281 block_cnt= 5 chunk_cnt= 2 mod=HashBuckConfCon [MEMORY] hold= 88,000 used= 56,704 count= 10 avg_used= 5,670 block_cnt= 9 chunk_cnt= 5 mod=BucketLock [MEMORY] hold= 81,920 used= 74,064 count= 2 avg_used= 37,032 block_cnt= 2 chunk_cnt= 2 mod=io_trace_map [MEMORY] hold= 79,328 used= 70,816 count= 8 avg_used= 8,852 block_cnt= 8 chunk_cnt= 5 mod=MallocInfoMap [MEMORY] hold= 73,728 used= 65,600 count= 1 avg_used= 65,600 block_cnt= 1 chunk_cnt= 1 mod=TCREF [MEMORY] hold= 73,728 used= 69,664 count= 1 avg_used= 69,664 block_cnt= 1 chunk_cnt= 1 mod=SuperBlockBuffe [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=ObLSLocation [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=KVCACHE_HAZARD [MEMORY] hold= 65,136 used= 63,096 count= 2 avg_used= 31,548 block_cnt= 2 chunk_cnt= 2 mod=ScheCacSysCacMa [MEMORY] hold= 64,000 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=RsEventQueue [MEMORY] hold= 57,344 used= 55,368 count= 1 avg_used= 55,368 block_cnt= 1 chunk_cnt= 1 mod=DepInfoTaskQ [MEMORY] hold= 56,128 used= 27,136 count= 377 avg_used= 71 block_cnt= 222 chunk_cnt= 22 mod=Coro [MEMORY] hold= 49,152 used= 32,768 count= 2 avg_used= 16,384 block_cnt= 2 chunk_cnt= 1 mod=CACHE_TNT_LST [MEMORY] hold= 49,152 used= 37,032 count= 3 avg_used= 12,344 block_cnt= 3 chunk_cnt= 2 mod=ReferedMap [MEMORY] hold= 49,152 used= 36,912 count= 2 avg_used= 18,456 block_cnt= 2 chunk_cnt= 2 mod=HashBuckSysConf [MEMORY] hold= 47,744 used= 38,400 count= 19 avg_used= 2,021 block_cnt= 15 chunk_cnt= 6 mod=TGTimer [MEMORY] hold= 45,760 used= 45,056 count= 11 avg_used= 4,096 block_cnt= 11 chunk_cnt= 6 mod=LinearHashMapCn [MEMORY] hold= 45,760 used= 45,056 count= 11 avg_used= 4,096 block_cnt= 11 chunk_cnt= 6 mod=LinearHashMapDi [MEMORY] hold= 41,264 used= 37,264 count= 2 avg_used= 18,632 block_cnt= 2 chunk_cnt= 1 mod=TaskRunnerSer [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdPoolMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SqlLoadData [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=KillSessMapBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=ClientSessBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=ObLongopsMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdConfMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucNamConMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HasBucConRefCoM [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucConPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucNamPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucSerUniMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucTenPooMa [MEMORY] hold= 40,960 used= 40,136 count= 1 avg_used= 40,136 block_cnt= 1 chunk_cnt= 1 mod=CommSysVarFac [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HasBucSerMigUnM [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucPooUniMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=T3MTBMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DDLSpeedCtrl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SessHoldMapBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TmpFileStoreMap [MEMORY] hold= 29,024 used= 26,512 count= 35 avg_used= 757 block_cnt= 25 chunk_cnt= 13 mod=ObGuard [MEMORY] hold= 24,944 used= 20,784 count= 2 avg_used= 10,392 block_cnt= 2 chunk_cnt= 2 mod=SqlNio [MEMORY] hold= 24,672 used= 24,264 count= 6 avg_used= 4,044 block_cnt= 6 chunk_cnt= 2 mod=ScheObSchemAren [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=FuncRuleMap [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=ServerCkptSlogH [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=GrpIdNameMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=leakMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=GrpNameIdMap [MEMORY] hold= 24,224 used= 21,672 count= 2 avg_used= 10,836 block_cnt= 2 chunk_cnt= 1 mod=SchemaStatuMap [MEMORY] hold= 23,904 used= 5,472 count= 228 avg_used= 24 block_cnt= 142 chunk_cnt= 24 mod=[T]MemoryContex [MEMORY] hold= 21,504 used= 20,352 count= 16 avg_used= 1,272 block_cnt= 16 chunk_cnt= 2 mod=IO_GROUP_MAP [MEMORY] hold= 18,416 used= 18,144 count= 4 avg_used= 4,536 block_cnt= 4 chunk_cnt= 3 mod=DeviceMng [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=HasBucTimZonInM [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MemDumpMap [MEMORY] hold= 16,384 used= 10,592 count= 1 avg_used= 10,592 block_cnt= 1 chunk_cnt= 1 mod=TenCompProgMgr [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ServerLogPool [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TbltRefreshMap [MEMORY] hold= 16,384 used= 8,992 count= 1 avg_used= 8,992 block_cnt= 1 chunk_cnt= 1 mod=IO_HEALTH [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=GenSchemVersMap [MEMORY] hold= 16,384 used= 12,296 count= 1 avg_used= 12,296 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 16,384 used= 9,336 count= 1 avg_used= 9,336 block_cnt= 1 chunk_cnt= 1 mod=InnerLobHash [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=LinkArray [MEMORY] hold= 16,032 used= 15,904 count= 2 avg_used= 7,952 block_cnt= 2 chunk_cnt= 1 mod=UpgProcSet [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=CommSysVarDefVa [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=SqlSession [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=MacroIdList [MEMORY] hold= 11,712 used= 8,944 count= 43 avg_used= 208 block_cnt= 26 chunk_cnt= 4 mod=Scheduler [MEMORY] hold= 10,816 used= 9,984 count= 13 avg_used= 768 block_cnt= 13 chunk_cnt= 4 mod=timer [MEMORY] hold= 9,472 used= 8,384 count= 16 avg_used= 524 block_cnt= 4 chunk_cnt= 2 mod=RpcBuffer [MEMORY] hold= 8,384 used= 8,104 count= 4 avg_used= 2,026 block_cnt= 3 chunk_cnt= 2 mod=TenantInfo [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PrepareOk [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ObTsTenantInfoN [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RetryTask [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=AddrList [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=Node [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SchemaRowKey [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PlanVaIdx [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=followerRenew [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=acceptOK [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerBlacklist [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerRegioMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerIdcMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerCidMap [MEMORY] hold= 7,968 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=LuaHandler [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HasNodTzInfM [MEMORY] hold= 7,776 used= 5,832 count= 27 avg_used= 216 block_cnt= 12 chunk_cnt= 8 mod=ObFuture [MEMORY] hold= 6,704 used= 3,840 count= 40 avg_used= 96 block_cnt= 30 chunk_cnt= 9 mod=PThread [MEMORY] hold= 5,424 used= 5,288 count= 2 avg_used= 2,644 block_cnt= 2 chunk_cnt= 2 mod=DeadLock [MEMORY] hold= 5,376 used= 5,248 count= 2 avg_used= 2,624 block_cnt= 1 chunk_cnt= 1 mod=RootContext [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=RebuildCtx [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=SqlPx [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmReq [MEMORY] hold= 4,592 used= 3,426 count= 16 avg_used= 214 block_cnt= 13 chunk_cnt= 9 mod=EventReHolder [MEMORY] hold= 3,888 used= 3,816 count= 1 avg_used= 3,816 block_cnt= 1 chunk_cnt= 1 mod=RecScheHisMap [MEMORY] hold= 3,888 used= 3,816 count= 1 avg_used= 3,816 block_cnt= 1 chunk_cnt= 1 mod=RemMasterMap [MEMORY] hold= 3,680 used= 3,360 count= 5 avg_used= 672 block_cnt= 5 chunk_cnt= 4 mod=[T]ObWarningBuf [MEMORY] hold= 3,264 used= 3,200 count= 1 avg_used= 3,200 block_cnt= 1 chunk_cnt= 1 mod=TenantTZ [MEMORY] hold= 2,768 used= 2,704 count= 1 avg_used= 2,704 block_cnt= 1 chunk_cnt= 1 mod=LoggerAlloc [MEMORY] hold= 2,544 used= 1,392 count= 16 avg_used= 87 block_cnt= 15 chunk_cnt= 8 mod=ObFunction [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=StorageS3 [MEMORY] hold= 1,728 used= 1,536 count= 3 avg_used= 512 block_cnt= 3 chunk_cnt= 2 mod=TntResourceMgr [MEMORY] hold= 1,600 used= 1,536 count= 1 avg_used= 1,536 block_cnt= 1 chunk_cnt= 1 mod=LobManager [MEMORY] hold= 1,504 used= 1,432 count= 1 avg_used= 1,432 block_cnt= 1 chunk_cnt= 1 mod=GtsRequestRpc [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=T3MQSyncLock [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=TsMgr [MEMORY] hold= 1,472 used= 1,400 count= 1 avg_used= 1,400 block_cnt= 1 chunk_cnt= 1 mod=GtiRequestRpc [MEMORY] hold= 1,408 used= 1,344 count= 1 avg_used= 1,344 block_cnt= 1 chunk_cnt= 1 mod=SchemaService [MEMORY] hold= 1,392 used= 1,328 count= 1 avg_used= 1,328 block_cnt= 1 chunk_cnt= 1 mod=GtsRpcProxy [MEMORY] hold= 1,392 used= 1,328 count= 1 avg_used= 1,328 block_cnt= 1 chunk_cnt= 1 mod=GtiRpcProxy [MEMORY] hold= 1,360 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=TENANT_PLAN_MAP [MEMORY] hold= 1,360 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IO_CHANNEL_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=INGRESS_MAP [MEMORY] hold= 1,184 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=memdumpqueue [MEMORY] hold= 1,168 used= 1,092 count= 1 avg_used= 1,092 block_cnt= 1 chunk_cnt= 1 mod=LogRegionMap [MEMORY] hold= 768 used= 704 count= 1 avg_used= 704 block_cnt= 1 chunk_cnt= 1 mod=ScheMgrCacheMap [MEMORY] hold= 640 used= 280 count= 5 avg_used= 56 block_cnt= 5 chunk_cnt= 1 mod=Log [MEMORY] hold= 528 used= 264 count= 3 avg_used= 88 block_cnt= 3 chunk_cnt= 2 mod=DestKAState [MEMORY] hold= 432 used= 272 count= 2 avg_used= 136 block_cnt= 2 chunk_cnt= 2 mod=unknown [MEMORY] hold= 288 used= 84 count= 3 avg_used= 28 block_cnt= 2 chunk_cnt= 2 mod=KeepAliveServer [MEMORY] hold= 240 used= 48 count= 3 avg_used= 16 block_cnt= 3 chunk_cnt= 3 mod=CreateEntity [MEMORY] hold= 224 used= 160 count= 1 avg_used= 160 block_cnt= 1 chunk_cnt= 1 mod=OccamTimeGuard [MEMORY] hold= 128 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=PxTargetMgr [MEMORY] hold= 80 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=Compressor [MEMORY] hold= 80 used= 7 count= 1 avg_used= 7 block_cnt= 1 chunk_cnt= 1 mod=SqlExpr [MEMORY] hold= 181,620,464 used= 178,203,526 count= 4,460 avg_used= 39,955 mod=SUMMARY [2024-04-22 16:23:15.132422] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=108] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 6,291,456 used= 2,658,560 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,600,016 used= 675,052 count= 12,703 avg_used= 53 block_cnt= 209 chunk_cnt= 2 mod=Buffer [MEMORY] hold= 1,005,024 used= 704,055 count= 3,307 avg_used= 212 block_cnt= 200 chunk_cnt= 3 mod=glibc_malloc [MEMORY] hold= 53,520 used= 37,729 count= 229 avg_used= 164 block_cnt= 29 chunk_cnt= 1 mod=S3SDK [MEMORY] hold= 2,658,560 used= 1,416,836 count= 16,239 avg_used= 87 mod=SUMMARY [2024-04-22 16:23:15.132465] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 115,343,360 used= 109,412,352 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 109,412,352 used= 109,208,832 count= 212 avg_used= 515,136 block_cnt= 212 chunk_cnt= 55 mod=CoStack [MEMORY] hold= 109,412,352 used= 109,208,832 count= 212 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:15.132484] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 6,291,456 used= 3,882,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,882,960 used= 3,662,624 count= 265 avg_used= 13,821 block_cnt= 47 chunk_cnt= 3 mod=OB_TEST2_PCODE [MEMORY] hold= 3,882,960 used= 3,662,624 count= 265 avg_used= 13,821 mod=SUMMARY [2024-04-22 16:23:15.132517] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 12,582,912 used= 12,484,608 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 8,323,072 used= 8,318,976 count= 4 avg_used= 2,079,744 block_cnt= 4 chunk_cnt= 4 mod=Logger [MEMORY] hold= 4,161,536 used= 4,159,488 count= 2 avg_used= 2,079,744 block_cnt= 2 chunk_cnt= 2 mod=ErrorLogger [MEMORY] hold= 12,484,608 used= 12,478,464 count= 6 avg_used= 2,079,744 mod=SUMMARY [2024-04-22 16:23:15.132576] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 33,783,808 used= 30,848,016 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 25,482,256 used= 25,332,400 count= 45 avg_used= 562,942 block_cnt= 23 chunk_cnt= 12 mod=DEFAULT [MEMORY] hold= 2,760,704 used= 2,750,496 count= 21 avg_used= 130,976 block_cnt= 21 chunk_cnt= 4 mod=PKTS_INBUF [MEMORY] hold= 1,835,008 used= 1,833,664 count= 14 avg_used= 130,976 block_cnt= 14 chunk_cnt= 4 mod=PKTC_INBUF [MEMORY] hold= 245,760 used= 244,320 count= 15 avg_used= 16,288 block_cnt= 15 chunk_cnt= 4 mod=SERVER_CTX_CHUN [MEMORY] hold= 196,608 used= 195,456 count= 12 avg_used= 16,288 block_cnt= 12 chunk_cnt= 3 mod=SERVER_RESP_CHU [MEMORY] hold= 196,608 used= 195,456 count= 12 avg_used= 16,288 block_cnt= 12 chunk_cnt= 3 mod=CLIENT_CB_CHUNK [MEMORY] hold= 131,072 used= 130,304 count= 8 avg_used= 16,288 block_cnt= 8 chunk_cnt= 2 mod=CLIENT_REQ_CHUN [MEMORY] hold= 30,848,016 used= 30,682,096 count= 127 avg_used= 241,591 mod=SUMMARY [2024-04-22 16:23:15.132669] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 11,292,672 used= 9,760,320 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 7,098,368 used= 7,078,824 count= 1 avg_used= 7,078,824 block_cnt= 1 chunk_cnt= 1 mod=SchemaIdVersion [MEMORY] hold= 2,088,768 used= 2,087,680 count= 2 avg_used= 1,043,840 block_cnt= 2 chunk_cnt= 2 mod=TenantSchemMgr [MEMORY] hold= 360,448 used= 327,680 count= 4 avg_used= 81,920 block_cnt= 4 chunk_cnt= 1 mod=SchemaMgrCache [MEMORY] hold= 110,208 used= 102,397 count= 5 avg_used= 20,479 block_cnt= 3 chunk_cnt= 1 mod=SchemaSysCache [MEMORY] hold= 31,872 used= 31,616 count= 4 avg_used= 7,904 block_cnt= 4 chunk_cnt= 1 mod=ScheTenaInfoVec [MEMORY] hold= 16,320 used= 16,064 count= 4 avg_used= 4,016 block_cnt= 4 chunk_cnt= 1 mod=SchemaSysVariab [MEMORY] hold= 15,936 used= 15,808 count= 2 avg_used= 7,904 block_cnt= 2 chunk_cnt= 1 mod=ScheTablInfoVec [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheIndeNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheTablIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=HiddenTblNames [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheRoutIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheRoutNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheTablNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=SchePackIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=SchePackNameMap [MEMORY] hold= 1,536 used= 1,024 count= 8 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSeCompo [MEMORY] hold= 1,152 used= 768 count= 6 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSePolic [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=ScheUdtNameMap [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=ScheUdtIdMap [MEMORY] hold= 1,152 used= 768 count= 6 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSeLabel [MEMORY] hold= 800 used= 512 count= 4 avg_used= 128 block_cnt= 3 chunk_cnt= 1 mod=SchemaSynonym [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=DBLINK_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabSeUserLe [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 4 chunk_cnt= 1 mod=DIRECTORY_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_POLICY_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlSqlMap [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_GROUP_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaProfile [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_CONTEXT_MGR [MEMORY] hold= 656 used= 584 count= 1 avg_used= 584 block_cnt= 1 chunk_cnt= 1 mod=TenSchMemMgrFoL [MEMORY] hold= 656 used= 584 count= 1 avg_used= 584 block_cnt= 1 chunk_cnt= 1 mod=TenaScheMemMgr [MEMORY] hold= 416 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheDataNameMap [MEMORY] hold= 416 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=MockFkParentTab [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaSecurAudi [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheAuxVpNameVe [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheForKeyNamMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaContext [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheConsNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlIdMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchePriTabPriMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchePriObjPriMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheTrigIdMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaTablespac [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheTrigNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaUdf [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaSequence [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaKeystore [MEMORY] hold= 9,760,320 used= 9,691,189 count= 134 avg_used= 72,322 mod=SUMMARY [2024-04-22 16:23:15.132722] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=29] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 201,011,200 used= 199,004,208 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 134,238,288 used= 134,217,736 count= 2 avg_used= 67,108,868 block_cnt= 2 chunk_cnt= 2 mod=CACHE_MAP_BKT [MEMORY] hold= 18,051,072 used= 18,033,840 count= 1 avg_used= 18,033,840 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MB_HANDLE [MEMORY] hold= 17,329,216 used= 8,968,960 count= 1,093 avg_used= 8,205 block_cnt= 1,093 chunk_cnt= 9 mod=StorageLoggerM [MEMORY] hold= 16,806,784 used= 16,786,208 count= 3 avg_used= 5,595,402 block_cnt= 3 chunk_cnt= 2 mod=FixeSizeBlocAll [MEMORY] hold= 6,311,936 used= 6,291,472 count= 1 avg_used= 6,291,472 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MAP_LOCK [MEMORY] hold= 3,709,312 used= 3,295,616 count= 114 avg_used= 28,908 block_cnt= 55 chunk_cnt= 4 mod=OccamThreadPool [MEMORY] hold= 2,543,616 used= 2,523,256 count= 1 avg_used= 2,523,256 block_cnt= 1 chunk_cnt= 1 mod=TenantConfig [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=BlockMap [MEMORY] hold= 5,984 used= 1,088 count= 68 avg_used= 16 block_cnt= 2 chunk_cnt= 2 mod=ConfigChecker [MEMORY] hold= 199,004,208 used= 190,126,112 count= 1,284 avg_used= 148,073 mod=SUMMARY [2024-04-22 16:23:15.132748] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant: 508, limit: 1,073,741,824 hold: 40,427,520 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 8,970,240 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 31,457,280 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:15.132803] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=32] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 8,970,240 used= 7,859,136 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4,423,680 used= 4,132,320 count= 60 avg_used= 68,872 block_cnt= 60 chunk_cnt= 3 mod=[T]ObSessionDIB [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 28,800 used= 24,960 count= 60 avg_used= 416 block_cnt= 7 chunk_cnt= 2 mod=OMT_Worker [MEMORY] hold= 16,320 used= 7,680 count= 120 avg_used= 64 block_cnt= 7 chunk_cnt= 2 mod=Coro [MEMORY] hold= 5,888 used= 1,440 count= 60 avg_used= 24 block_cnt= 7 chunk_cnt= 2 mod=[T]MemoryContex [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 7,859,136 used= 7,522,400 count= 304 avg_used= 24,744 mod=SUMMARY [2024-04-22 16:23:15.132874] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 31,457,280 used= 30,965,760 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 30,965,760 used= 30,908,160 count= 60 avg_used= 515,136 block_cnt= 60 chunk_cnt= 15 mod=CoStack [MEMORY] hold= 30,965,760 used= 30,908,160 count= 60 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:15.133011] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant: 509, limit: 1,073,741,824 hold: 25,747,456 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 6,873,088 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 18,874,368 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:15.133039] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 6,873,088 used= 5,994,736 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,580,480 used= 2,410,520 count= 35 avg_used= 68,872 block_cnt= 35 chunk_cnt= 2 mod=[T]ObSessionDIB [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 16,800 used= 14,560 count= 35 avg_used= 416 block_cnt= 4 chunk_cnt= 1 mod=OMT_Worker [MEMORY] hold= 9,648 used= 4,480 count= 70 avg_used= 64 block_cnt= 4 chunk_cnt= 1 mod=Coro [MEMORY] hold= 3,360 used= 840 count= 35 avg_used= 24 block_cnt= 4 chunk_cnt= 1 mod=[T]MemoryContex [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 5,994,736 used= 5,786,400 count= 179 avg_used= 32,326 mod=SUMMARY [2024-04-22 16:23:15.133083] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 18,874,368 used= 18,063,360 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 18,063,360 used= 18,029,760 count= 35 avg_used= 515,136 block_cnt= 35 chunk_cnt= 9 mod=CoStack [MEMORY] hold= 18,063,360 used= 18,029,760 count= 35 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:15.133170] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant: 1, limit: 4,294,967,296 hold: 651,026,432 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 453,873,664 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 130,023,424 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 2,097,152 limit= 858,993,440 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 18,894,848 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33,554,432 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:15.133420] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 453,873,664 used= 379,141,328 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 31 [MEMORY] hold= 157,024,256 used= 156,961,472 count= 39 avg_used= 4,024,653 block_cnt= 39 chunk_cnt= 39 mod=MysqlRequesReco [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 23,678,976 used= 23,614,912 count= 23 avg_used= 1,026,735 block_cnt= 23 chunk_cnt= 12 mod=IoControl [MEMORY] hold= 16,662,528 used= 15,565,072 count= 226 avg_used= 68,872 block_cnt= 226 chunk_cnt= 32 mod=[T]ObSessionDIB [MEMORY] hold= 14,483,248 used= 14,360,745 count= 93 avg_used= 154,416 block_cnt= 33 chunk_cnt= 7 mod=OmtTenant [MEMORY] hold= 8,986,624 used= 8,945,664 count= 2 avg_used= 4,472,832 block_cnt= 2 chunk_cnt= 2 mod=ServerObjecPool [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,257,536 used= 8,240,064 count= 2 avg_used= 4,120,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 5,234,688 used= 4,653,056 count= 71 avg_used= 65,536 block_cnt= 71 chunk_cnt= 9 mod=[T]char [MEMORY] hold= 4,943,872 used= 4,915,456 count= 2 avg_used= 2,457,728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4,571,136 used= 4,525,056 count= 6 avg_used= 754,176 block_cnt= 6 chunk_cnt= 5 mod=ResourceGroup [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 4,214,784 used= 4,194,336 count= 1 avg_used= 4,194,336 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 4 mod=SqlDtlQueue [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,864 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,166,784 used= 2,142,208 count= 2 avg_used= 1,071,104 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 2,080,768 used= 2,079,744 count= 1 avg_used= 2,079,744 block_cnt= 1 chunk_cnt= 1 mod=LogReplayTask [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 2 mod=DagMap [MEMORY] hold= 1,777,664 used= 1,770,696 count= 1 avg_used= 1,770,696 block_cnt= 1 chunk_cnt= 1 mod=TableProc [MEMORY] hold= 1,596,912 used= 1,588,632 count= 3 avg_used= 529,544 block_cnt= 3 chunk_cnt= 2 mod=TabletMap [MEMORY] hold= 1,584,288 used= 1,575,992 count= 3 avg_used= 525,330 block_cnt= 2 chunk_cnt= 1 mod=PartitTableTask [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1,269,184 used= 1,263,904 count= 6 avg_used= 210,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 786,560 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 3 mod=LogAlloc [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 655,360 used= 650,752 count= 1 avg_used= 650,752 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 446,976 used= 440,768 count= 97 avg_used= 4,544 block_cnt= 97 chunk_cnt= 17 mod=[T]ObTraceEvent [MEMORY] hold= 442,368 used= 438,432 count= 3 avg_used= 146,144 block_cnt= 3 chunk_cnt= 2 mod=SQLSessionInfo [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 403,648 used= 378,880 count= 258 avg_used= 1,468 block_cnt= 53 chunk_cnt= 3 mod=LSMap [MEMORY] hold= 385,024 used= 377,120 count= 1 avg_used= 377,120 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 365,040 used= 362,160 count= 45 avg_used= 8,048 block_cnt= 45 chunk_cnt= 11 mod=[T]ObPerfEventR [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 212,992 used= 212,160 count= 1 avg_used= 212,160 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 196,608 used= 196,224 count= 3 avg_used= 65,408 block_cnt= 3 chunk_cnt= 3 mod=StorMetaCacheIO [MEMORY] hold= 152,000 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 9 mod=LSIter [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 78,408 count= 9 avg_used= 8,712 block_cnt= 9 chunk_cnt= 6 mod=[T]ObDSActionAr [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 100,008 count= 1 avg_used= 100,008 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 98,304 used= 96,032 count= 1 avg_used= 96,032 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 98,304 used= 97,408 count= 1 avg_used= 97,408 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 94,240 used= 90,112 count= 64 avg_used= 1,408 block_cnt= 17 chunk_cnt= 5 mod=TxCtxMgr [MEMORY] hold= 81,920 used= 65,536 count= 2 avg_used= 32,768 block_cnt= 2 chunk_cnt= 1 mod=CsDictPl [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 72,240 used= 1,024 count= 896 avg_used= 1 block_cnt= 10 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 64,624 used= 31,376 count= 455 avg_used= 68 block_cnt= 125 chunk_cnt= 32 mod=Coro [MEMORY] hold= 56,448 used= 55,584 count= 12 avg_used= 4,632 block_cnt= 12 chunk_cnt= 5 mod=WorkerMap [MEMORY] hold= 49,152 used= 48,576 count= 1 avg_used= 48,576 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 48,832 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=DictPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=RawPl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=ConstPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=CsIntPl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=CsStrPl [MEMORY] hold= 40,000 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 36,064 used= 32,928 count= 49 avg_used= 672 block_cnt= 20 chunk_cnt= 8 mod=[T]ObWarningBuf [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_ClientTask [MEMORY] hold= 32,768 used= 17,408 count= 2 avg_used= 8,704 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTTP [MEMORY] hold= 32,768 used= 25,600 count= 1 avg_used= 25,600 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 32,576 used= 26,079 count= 2 avg_used= 13,039 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 25,440 used= 22,048 count= 53 avg_used= 416 block_cnt= 14 chunk_cnt= 6 mod=OMT_Worker [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=IntDiffPl [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=RlePl [MEMORY] hold= 22,160 used= 5,424 count= 226 avg_used= 24 block_cnt= 91 chunk_cnt= 26 mod=[T]MemoryContex [MEMORY] hold= 17,472 used= 16,128 count= 21 avg_used= 768 block_cnt= 18 chunk_cnt= 6 mod=TGTimer [MEMORY] hold= 16,800 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 3 chunk_cnt= 3 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 13,696 count= 1 avg_used= 13,696 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 12,288 count= 1 avg_used= 12,288 block_cnt= 1 chunk_cnt= 1 mod=DiskIteratorSto [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=StrDiffPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=HexStrPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ColEqualPl [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxIMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_TableCtxMgr [MEMORY] hold= 15,744 used= 15,616 count= 2 avg_used= 7,808 block_cnt= 2 chunk_cnt= 2 mod=HashNodeLCSta [MEMORY] hold= 8,320 used= 8,192 count= 2 avg_used= 4,096 block_cnt= 2 chunk_cnt= 2 mod=LinearHashMapCn [MEMORY] hold= 8,320 used= 8,192 count= 2 avg_used= 4,096 block_cnt= 2 chunk_cnt= 2 mod=LinearHashMapDi [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTableMgr [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=TabletIDSetNode [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePlanCac [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoNode [MEMORY] hold= 7,856 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,856 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletNode [MEMORY] hold= 5,184 used= 5,120 count= 1 avg_used= 5,120 block_cnt= 1 chunk_cnt= 1 mod=CtxBlockPl [MEMORY] hold= 4,800 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=CaLsInfoMap [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,160 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=CsCtxBlockPl [MEMORY] hold= 4,160 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 3,072 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,336 used= 1,728 count= 8 avg_used= 216 block_cnt= 4 chunk_cnt= 3 mod=LogIOTask [MEMORY] hold= 1,984 used= 1,920 count= 1 avg_used= 1,920 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 1,872 used= 1,800 count= 1 avg_used= 1,800 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,872 used= 1,296 count= 9 avg_used= 144 block_cnt= 5 chunk_cnt= 4 mod=ObTxLSLogCb [MEMORY] hold= 1,664 used= 1,600 count= 1 avg_used= 1,600 block_cnt= 1 chunk_cnt= 1 mod=LogPartFetCtxPo [MEMORY] hold= 1,616 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,488 used= 1,352 count= 2 avg_used= 676 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=LockMemtableMgr [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,184 used= 1,056 count= 2 avg_used= 528 block_cnt= 1 chunk_cnt= 1 mod=PoolArenaArray [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,120 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 848 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=TabShellTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=DiskUsageTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=FlushTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=CheckPointTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=TLD_TIMER [MEMORY] hold= 720 used= 648 count= 1 avg_used= 648 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 624 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 544 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 480 used= 120 count= 5 avg_used= 24 block_cnt= 3 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 336 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 192 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 192 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 176 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 176 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 160 used= 16 count= 2 avg_used= 8 block_cnt= 2 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 128 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 112 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 80 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 379,141,328 used= 372,914,203 count= 28,353 avg_used= 13,152 mod=SUMMARY [2024-04-22 16:23:15.133530] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=97] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 4,194,304 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2024-04-22 16:23:15.133549] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 794,624 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 794,624 used= 788,032 count= 1 avg_used= 788,032 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 794,624 used= 788,032 count= 1 avg_used= 788,032 mod=SUMMARY [2024-04-22 16:23:15.133569] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 2,097,152 used= 1,286,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,232,192 used= 1,115,344 count= 43 avg_used= 25,938 block_cnt= 43 chunk_cnt= 1 mod=SqlPhyPlan [MEMORY] hold= 32,000 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 1 mod=SqlPlanCache [MEMORY] hold= 20,608 used= 19,840 count= 10 avg_used= 1,984 block_cnt= 3 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 1,728 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 1,286,528 used= 1,168,528 count= 59 avg_used= 19,805 mod=SUMMARY [2024-04-22 16:23:15.133590] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,848 used= 59,018 count= 263 avg_used= 224 block_cnt= 20 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,944 used= 59,050 count= 264 avg_used= 223 mod=SUMMARY [2024-04-22 16:23:15.133608] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 130,023,424 used= 126,443,520 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 126,443,520 used= 126,208,320 count= 245 avg_used= 515,136 block_cnt= 245 chunk_cnt= 62 mod=CoStack [MEMORY] hold= 126,443,520 used= 126,208,320 count= 245 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:15.133640] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 2,097,152 used= 466,944 limit= 858,993,440 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 401,408 used= 400,064 count= 2 avg_used= 200,032 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=N_TabletPool [MEMORY] hold= 466,944 used= 465,472 count= 3 avg_used= 155,157 mod=SUMMARY [2024-04-22 16:23:15.133661] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 65,536 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=LobAllocator [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 mod=SUMMARY [2024-04-22 16:23:15.133690] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=5] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 18,894,848 used= 17,870,464 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17,862,464 used= 17,833,792 count= 3 avg_used= 5,944,597 block_cnt= 3 chunk_cnt= 2 mod=MEMTABLE_ARENA [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 17,870,464 used= 17,841,728 count= 4 avg_used= 4,460,432 mod=SUMMARY [2024-04-22 16:23:15.133745] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33,554,432 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 33,554,432 free_size= 33,554,432 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2024-04-22 16:23:15.133793] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:122) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=12] ====== tenants memory info ====== === TENANTS MEMORY INFO === divisive_memory_used= 85,983,232 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 579,035,136 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 40,427,520 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 25,747,456 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,713,774,195,133,158 active_memstore_used= 0 total_memstore_used= 0 total_memstore_hold= 0 memstore_freeze_trigger_limit= 343,597,360 memstore_limit= 1,717,986,880 mem_tenant_limit= 4,294,967,296 mem_tenant_hold= 651,026,432 max_mem_memstore_can_get_now= 1,821,970,400 memstore_alloc_pos= 0 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2024-04-22 16:23:15.132531] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=867] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:3, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1713773681580810, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=9, offset=1469}, ls_meta_entry:[5](ver=0,mode=0,seq=4), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:53, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:232379764, recv_hp_rpc_cnt:1, recv_np_rpc_cnt:958, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:1379685, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=15443 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:2, queue_size:0, recv_req_cnt:2138, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773684733335}{group_id:10, queue_size:0, recv_req_cnt:1345, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773689875802}{group_id:5, queue_size:0, recv_req_cnt:813, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773687232900}{group_id:19, queue_size:0, recv_req_cnt:40, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1713774180915685}{group_id:9, queue_size:0, recv_req_cnt:12101, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773677273264}{group_id:1, queue_size:0, recv_req_cnt:5731, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1713774144054964}, rpc_stat_info: pcode=0x14a:cnt=89 pcode=0x288:cnt=44 pcode=0x502:cnt=29 pcode=0x4a9:cnt=12 pcode=0x710:cnt=12, token_change_ts:1713773687300994, tenant_role:1}) [2024-04-22 16:23:15.134015] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:162) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=12] [CHUNK_MGR] free=14 pushes=61968 pops=61954 limit= 26,843,545,600 hold= 1,329,823,744 total_hold= 1,421,869,056 used= 1,296,236,544 freelist_hold= 33,587,200 large_freelist_hold= 4,227,072 maps= 434 unmaps= 12 large_maps= 34 large_unmaps= 5 huge_maps= 32 huge_unmaps= 3 memalign=0 resident_size= 1,769,279,488 virtual_memory_used= 2,297,659,392 [2024-04-22 16:23:15.134134] INFO print (ob_malloc_time_monitor.cpp:39) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=11] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 0, 10): delta_total_cost_time= 14459, delta_count= 8894, avg_cost_time= 1 [MALLOC_TIME_MONITOR] [ 10, 100): delta_total_cost_time= 5371, delta_count= 171, avg_cost_time= 31 [MALLOC_TIME_MONITOR] [ 100, 1000): delta_total_cost_time= 1671, delta_count= 9, avg_cost_time= 185 [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2024-04-22 16:23:15.133973] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1437] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:5, max_cpu:5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713773672949917, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"5.000000000000000000e+00", unit_max_cpu:"5.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:52, max_worker_cnt:150, stopped:0, worker_us:93133956, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2317704, workers:52, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1713773687322359, tenant_role:0}) [2024-04-22 16:23:15.139895] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=5919] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:2.5, max_cpu:2.5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713773672930258, is_removed:false}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"2.500000000000000000e+00", unit_max_cpu:"2.500000000000000000e+00", total_worker_cnt:35, min_worker_cnt:27, max_worker_cnt:150, stopped:0, worker_us:70235975, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:1158300, workers:27, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1713773687342553, tenant_role:0}) [2024-04-22 16:23:15.163001] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.163142] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=39000, remain_us=1045268, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.168496] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.168613] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] will sleep(sleep_us=73000, remain_us=19658530, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=73, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.177142] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:588) [10807][T1_TntSharedTim][T1][Y0-0000000000000000-0-0] [lt=8] dump dtl interm result cost(us)(dump_cost=4175, ret=0, interm count=0, dump count=51) [2024-04-22 16:23:15.181854] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:603) [10807][T1_TntSharedTim][T1][Y0-0000000000000000-0-0] [lt=68] clear dtl interm result cost(us)(clear_cost=4639, ret=0, gc_.expire_keys_.count()=0, dump count=51, interm count=0, clean count=0) [2024-04-22 16:23:15.190311] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=50] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.190337] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=72] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.190395] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=69] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.190490] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.190568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.190599] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.191160] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.191359] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.191740] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.191922] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=60] gc stale ls task succ [2024-04-22 16:23:15.192171] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.192311] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=95] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10514, mem_used=79030272) [2024-04-22 16:23:15.192412] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=34] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:15.192638] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=52] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.192919] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=55] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.192977] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=55] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:15.193128] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=49] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.193362] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.193674] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=65] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.193681] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.193758] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.193970] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=50] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.193993] INFO pn_ratelimit (group.c:589) [10608][IngressService][T0][Y0-0000000000000000-0-0] [lt=47] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:23:15.194659] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC65-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.194690] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC65-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:15.194704] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC65-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:15.194970] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=98] current monitor number(seq_=-1) [2024-04-22 16:23:15.195550] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC66-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.195581] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC66-0-0] [lt=24] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:15.195643] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:15.195704] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:15.195755] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:15.195805] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:15.196003] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=46] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.196031] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.196051] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.197185] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=74] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.197955] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.198548] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC50-0-0] [lt=21][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.198618] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC50-0-0] [lt=50] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:15.198648] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC50-0-0] [lt=27][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:15.198875] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] ls blacklist refresh finish(cost_time=2447) [2024-04-22 16:23:15.199041] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=92] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:15.199771] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=59] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:15.200581] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=96] ====== tenant freeze timer task ====== [2024-04-22 16:23:15.201668] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=74][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:126, tid:10410}]) [2024-04-22 16:23:15.202812] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=13][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.202841] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=22] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:15.202854] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:15.202861] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:15.202867] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:15.202873] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:15.202878] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:15.202883] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:15.202887] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:15.202899] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=12][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:15.202903] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:15.202907] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:15.202911] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:15.202915] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:15.202920] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:15.202929] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:15.202933] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.202938] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.202942] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:15.202947] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:15.202955] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:15.202961] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.202978] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:15.202993] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.202997] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.203000] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:15.203009] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:15.202983] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1713774195202979, bytes: 66170511, bw: 0.036995 MB/s, add_ts: 1000336, add_bytes: 38805 [2024-04-22 16:23:15.203015] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:15.203021] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:15.203025] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:15.203029] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:15.203034] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1484050, start=1713774195202713, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:15.203044] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=9][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:15.203048] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=3][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:15.203086] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=5][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:15.203090] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED82-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:15.205608] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=49] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713774194704094], gts:1713774197447619589, latest_srr:[mts=1713774195205543]}) [2024-04-22 16:23:15.206023] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.206131] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=1002279, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.209258] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=49] PNIO [ratelimit] time: 1713774195209255, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007596, add_bytes: 0 [2024-04-22 16:23:15.209653] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=8] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:15.211630] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=125] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:15.211979] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=254][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:15.212048] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:15.212078] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:15.212135] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=54][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:15.212159] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=22][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:15.212178] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:15.212191] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:15.212207] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:15.212219] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:15.212231] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:15.212221] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=86] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:15.212242] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:15.212253] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:15.212264] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:15.212284] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=18][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:15.212296] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:15.212318] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:15.212331] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.212345] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:15.212356] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:15.212367] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:15.212382] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:15.212393] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.212418] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:15.212491] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=65][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.212506] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:15.212524] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=17][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:15.212558] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:15.212595] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=34][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:15.212619] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=21][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:15.212636] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=15][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:15.212648] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:15.212661] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd148e050, start=1713774195211721, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:15.212679] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:15.212691] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:15.212780] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=14][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:15.212815] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=31][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:15.212849] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=31][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.212868] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=18][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.212890] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=17][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:15.212914] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=19][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:15.212931] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7E-0-0] [lt=16][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:15.221587] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10483][LSSysTblUp0][T0][YB42AC175881-000616AB0D0FFD32-0-0] [lt=32][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:15.240730] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C358B-0-0] [lt=476][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:15.240844] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C358B-0-0] [lt=109][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:15.240912] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C358B-0-0] [lt=64][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:15.240955] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C358B-0-0] [lt=41][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:15.245503] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.245593] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=74000, remain_us=19581549, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=74, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.246320] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=40][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:15.249622] INFO [PALF] try_reset_last_fetch_log_info_ (log_sliding_window.cpp:1837) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C22-0-0] [lt=64] reset last fetch log info(palf_id=1, self="172.23.88.129:2882", expected_end_lsn={lsn:42301784}) [2024-04-22 16:23:15.249792] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C22-0-0] [lt=60][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2953, log_task={header:{begin_lsn:{lsn:359970}, end_lsn:{lsn:360083}, log_id:2953, min_scn:{val:1713774197406167741, v:0}, max_scn:{val:1713774197406167741, v:0}, data_len:57, proposal_id:8, prev_lsn:{lsn:359848}, prev_proposal_id:8, committed_end_lsn:{lsn:359970}, data_checksum:2031283194, accum_checksum:295676114, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774194303063, freeze_ts:1713774194303063, submit_ts:1713774194303066, flushed_ts:1713774194314275, gen_to_freeze cost time:0, gen_to_submit cost time:3, submit_to_flush cost time:11209}, fs_cb_begin_ts=1713774195249731, log_life_time=946668) [2024-04-22 16:23:15.250082] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.250193] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=41000, remain_us=958216, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.295605] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.295699] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=42000, remain_us=912710, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.299514] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299576] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299605] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299623] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299636] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299649] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299662] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299675] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299714] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299753] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299774] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299803] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299836] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299852] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299865] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.299878] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:15.301090] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:15.301160] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197320369684, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774195301077) [2024-04-22 16:23:15.301199] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774195100982, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:15.302020] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=52][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:500, tid:10796}]) [2024-04-22 16:23:15.319747] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:74, local_retry_times:74, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.319863] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=85][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.319885] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.319897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.319923] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.323768] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] [2024-04-22 16:23:15.323806] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.323834] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.323851] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.323872] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.323894] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.323907] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.323956] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.324028] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.324049] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:15.324075] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=75, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.324131] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38] will sleep(sleep_us=75000, remain_us=19503012, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=75, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.337856] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:42, local_retry_times:42, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.337939] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.337968] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.337981] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.338006] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.341731] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:15.341809] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=75][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.341907] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=95][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.341926] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.341941] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.341952] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.341963] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.342073] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=70] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.342112] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.342125] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:15.342142] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=43, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.342176] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19] will sleep(sleep_us=43000, remain_us=866234, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.385342] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=47][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:43, local_retry_times:43, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.385491] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=124][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.385523] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.385543] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.385588] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.389505] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:15.389541] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.389569] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.389591] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.389613] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.389632] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.389650] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.389704] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.389798] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=86][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.389819] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:15.389845] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=44, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.389895] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32] will sleep(sleep_us=44000, remain_us=818515, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.399283] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:75, local_retry_times:75, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.399378] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.399407] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.399426] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.399507] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.405156] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] [2024-04-22 16:23:15.405204] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.405252] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.405269] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.405283] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.405295] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.405306] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.405359] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.405414] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.405477] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:15.405510] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=76, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.405515] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFC46-0-0] [lt=49][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774195405324}) [2024-04-22 16:23:15.405548] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21] will sleep(sleep_us=76000, remain_us=19421595, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=76, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.411813] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:15.412805] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:15.426015] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=47] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=13159, clean_start_pos=4026528, clean_num=125829) [2024-04-22 16:23:15.434047] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:44, local_retry_times:44, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.434139] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.434178] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.434198] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.434233] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.437865] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] [2024-04-22 16:23:15.437926] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=58][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.437957] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.437981] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.438004] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.438018] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.438029] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.438077] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.438116] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.438130] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:15.438147] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=45, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.438181] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=45000, remain_us=770228, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.466317] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=106] BlockGCTimerTask success(ret=0, cost_time_us=34, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-04-22 16:23:15.481699] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:76, local_retry_times:76, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.481825] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=92][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.481848] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.481859] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.481886] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.483345] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=54][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:45, local_retry_times:45, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.483426] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.483465] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.483476] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.483530] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.486289] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:15.486341] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=49][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.486373] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.486392] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.486406] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.486478] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=69][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.486492] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.486551] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.486609] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=49][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.486632] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:15.486658] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=77, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.486705] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30] will sleep(sleep_us=77000, remain_us=19340437, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=77, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.487894] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] [2024-04-22 16:23:15.487923] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.487940] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.487953] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.487977] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.487988] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.488003] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.488035] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.488055] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.488062] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:15.488072] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=46, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.488090] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] will sleep(sleep_us=46000, remain_us=720319, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.501286] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=118][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:15.501340] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:15.501414] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197510469743, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774195501273) [2024-04-22 16:23:15.501480] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774195301318, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:15.534181] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:46, local_retry_times:46, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.534241] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=43][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.534252] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.534274] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.534290] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.538072] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] [2024-04-22 16:23:15.538108] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.538129] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.538147] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.538159] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.538166] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.538174] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.538217] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.538263] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.538275] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:15.538303] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=47, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.538326] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] will sleep(sleep_us=47000, remain_us=670083, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.563901] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:77, local_retry_times:77, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.563999] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.564027] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.564046] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.564098] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.568043] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:15.568090] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.568123] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.568141] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.568155] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.568167] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.568177] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.568227] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.568281] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.568303] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:15.568336] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=78, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.568436] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=79] will sleep(sleep_us=78000, remain_us=19258707, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=78, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.585462] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:47, local_retry_times:47, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.585553] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.585575] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.585586] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.585635] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.589132] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] [2024-04-22 16:23:15.589173] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.589201] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.589240] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=36][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.589263] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.589282] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.589295] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.589341] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.589380] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.589393] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:15.589411] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=48, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.589495] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=48000, remain_us=618915, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.599791] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:249) [10820][T1_RFLWorker][T1][YB42AC175881-000616AB070FEBAD-0-0] [lt=52] ObRemoteFetchWorker is running(thread_index=0) [2024-04-22 16:23:15.601914] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2644) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=171] [PALF_DUMP](palf_id=1, self="172.23.88.129:2882", [SlidingWindow]={palf_id:1, self:"172.23.88.129:2882", lsn_allocator:{max_log_id:2954, max_lsn:360205, max_scn:{val:1713774197510469743, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:360205}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:358576}}, last_submit_lsn:{lsn:360083}, last_submit_end_lsn:{lsn:360205}, last_submit_log_id:2954, last_submit_log_pid:8, max_flushed_lsn:{lsn:360083}, max_flushed_end_lsn:{lsn:360205}, max_flushed_log_pid:8, committed_end_lsn:{lsn:360205}, last_slide_log_id:2954, last_slide_scn:{val:1713774197510469743, v:0}, last_slide_lsn:{lsn:360083}, last_slide_end_lsn:{lsn:360205}, last_slide_log_pid:8, last_slide_log_accum_checksum:2310754929, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f3cd2de4530}, [StateMgr]={this:0x7f3cd2de8330, self:"172.23.88.129:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, leader:"172.23.88.130:2882", leader_epoch:11, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:358576}, scan_disk_log_finished:true, last_check_start_id:2955, is_changing_config_with_arb:false, reconfirm_start_time_us:1713774181765332, palf_role_change_cb:0x7f3cd2deebf0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.23.88.129:2882", alive_paxos_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_end_lsn:{lsn:356581}, prev_mode_pid:7}, reconfig_barrier:{prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_end_lsn:{lsn:356581}, prev_mode_pid:7}, persistent_config_version:{proposal_id:8, config_seq:9}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1713774193299233, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1713774193299233, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f3cd2de5c70}, [ModeMgr]={palf_id:1, self:"172.23.88.129:2882", applied_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:114688}, readable_log_tail:{lsn:114688}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66990080, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}, log_storage:{log_tail:{lsn:360205}, readable_log_tail:{lsn:360205}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/log", dir_fd:235, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66744563, block_header_serialize_buf_:0x7f3cd2dec138, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1713774181764894, this:0x7f3cd2de8a70}, [Reconfirm]={palf_id:1, self:"172.23.88.129:2882", state:"FINISHED", new_proposal_id:7, prepare_log_ack_list:1"172.23.88.131:2882";, curr_paxos_follower_list:2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.23.88.129:2882", majority_max_accept_pid:6, majority_max_lsn:{lsn:356581}, saved_end_lsn:{lsn:356581}, last_submit_prepare_req_time_us:1713774181765723, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f3cd2de84b0}) [2024-04-22 16:23:15.602516] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=586] [PALF_DUMP](palf_id=1, self="172.23.88.129:2882", ack_info_list=[{member:{server:"172.23.88.129:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.130:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.131:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}]) [2024-04-22 16:23:15.602631] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4287) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=104] current log_task status(palf_id=1, self="172.23.88.129:2882", log_id=2955, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.23.88.129:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.130:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.131:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}]) [2024-04-22 16:23:15.612105] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=63] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:15.626741] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:15.637731] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=55][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:48, local_retry_times:48, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.637861] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=106][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.637905] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=42][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.637917] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.637945] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.641909] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4725] [2024-04-22 16:23:15.641951] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.641981] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.642000] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.642015] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.642027] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.642038] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.642085] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.642140] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.642178] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=35][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:15.642197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=49, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.642231] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=49000, remain_us=566179, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.646644] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=77][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:78, local_retry_times:78, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.646734] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.646762] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.646775] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.646801] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.650633] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:15.650690] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=54][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.650724] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.650746] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.650812] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.650842] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.650860] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.650924] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.650996] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.651018] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:15.651053] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=79, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.651125] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=52] will sleep(sleep_us=79000, remain_us=19176019, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=79, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.691367] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.691464] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=53][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.691478] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.691486] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.691504] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.695551] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] [2024-04-22 16:23:15.695592] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:15.695626] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:15.695671] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=42][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:15.695690] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:15.695704] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:15.695720] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.695773] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.695837] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=55][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:15.695855] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:15.695875] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=50, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:15.695912] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21] will sleep(sleep_us=50000, remain_us=512498, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.698627] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:310) [10850][T1_ArcFetcher][T1][YB42AC175881-000616AB076FEBAD-0-0] [lt=55] ObArchiveFetcher is running(thread_index=0) [2024-04-22 16:23:15.701623] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:15.701689] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197510469743, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774195701610) [2024-04-22 16:23:15.701726] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774195501502, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:15.701815] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:15.707040] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=50] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:15.730313] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=42][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:79, local_retry_times:79, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:15.730410] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.730476] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:15.730513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:15.730541] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:15.734672] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.734760] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=80000, remain_us=19092383, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=80, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.736516] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C358E-0-0] [lt=187][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:15.736582] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C358E-0-0] [lt=62][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:15.736671] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C358E-0-0] [lt=84][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:15.736711] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C358E-0-0] [lt=39][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:15.748693] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.748780] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=51000, remain_us=459630, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.752704] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=97] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:15.802602] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.802696] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=52000, remain_us=405713, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.812301] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=60] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:15.820857] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.820946] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=81000, remain_us=19006196, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=81, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.827511] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=71] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:15.859908] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.860003] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=53000, remain_us=348406, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.901839] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:15.901892] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197510469743, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774195901826) [2024-04-22 16:23:15.901923] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774195701803, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:15.906269] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.906362] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=82000, remain_us=18920780, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=82, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:15.916883] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.916971] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=54000, remain_us=291439, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.975896] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.975991] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=55000, remain_us=232418, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:15.988543] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:15.992030] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:15.992155] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=2] will sleep(sleep_us=83000, remain_us=18834988, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=83, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.012577] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:16.028322] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=113] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:16.035212] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.035346] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=56000, remain_us=173064, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:16.078903] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.079013] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=84000, remain_us=18748130, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=84, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.094430] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.094520] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=57000, remain_us=113889, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774196208408) [2024-04-22 16:23:16.100231] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=60] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:16.101228] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED3E-0-0] [lt=18] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:16.102049] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:16.102131] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197510469743, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774196102035) [2024-04-22 16:23:16.102154] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774195901939, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:16.155666] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1][errcode=-4012] [2024-04-22 16:23:16.155703] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=35][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.155726] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:23:16.155739] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.155750] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:23:16.155757] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:23:16.155765] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:16.155773] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:16.155781] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.155810] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=58, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.155828] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:58, local_retry_times:58, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:23:16.155853] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:16.155861] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:16.155868] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:23:16.155884] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:23:16.155897] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1945575) [2024-04-22 16:23:16.155908] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:23:16.155921] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:16.155938] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:23:16.155947] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:16.155958] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:16.155976] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:16.156058] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:23:16.156081] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:16.156095] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:23:16.156110] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:16.156130] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:16.156146] WDIAG [SHARE.SCHEMA] operator() (ob_multi_version_schema_service.cpp:2425) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4012] fail to get baseline_schema_version(ret=-4012) [2024-04-22 16:23:16.156159] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2495) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11] [REFRESH_SCHEMA] end refresh and add schema(ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], cost=1947781) [2024-04-22 16:23:16.156196] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:308) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4012] fail to refresh and add schema(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:16.156210] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:314) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13] [REFRESH_SCHEMA] end refresh schema with new mode, (ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], used time=1947878) [2024-04-22 16:23:16.160264] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.160316] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] already timeout, do not need sleep(sleep_us=0, remain_us=1995916, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.163200] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.163239] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=1000, remain_us=1992993, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.166086] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.166170] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=85000, remain_us=18660973, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=85, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.168095] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.168139] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=2000, remain_us=1988093, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.172691] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.172738] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=3000, remain_us=1983493, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.179552] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.179619] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=4000, remain_us=1976613, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.186734] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.186791] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=5000, remain_us=1969441, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.190478] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=59] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.190559] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.190729] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.190729] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.190740] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=52] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.190793] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=44] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.191249] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.191477] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.191510] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.191864] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.192039] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=19] gc stale ls task succ [2024-04-22 16:23:16.192554] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:171) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=59] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1, queue_size=10000000, config_mem_limit_=128849018, request_manager_->get_size_used()=10557, evict_high_size_level=9000000, evict_low_size_level=8000000, allocator->allocated()=79030272, evict_high_mem_level=103079214, evict_low_mem_level=77309410) [2024-04-22 16:23:16.192665] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.192677] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=119] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10557, mem_used=79030272) [2024-04-22 16:23:16.192700] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=85] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:16.192722] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.192846] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.192853] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.192927] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.193162] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=57] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:16.193214] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=99] ====== check clog disk timer task ====== [2024-04-22 16:23:16.193249] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=27] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:16.193312] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=41] cannot_recycle_log_size statistics(cannot_recycle_log_size=360205, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:16.193676] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.194474] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.194825] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.195106] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.195157] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=6000, remain_us=1961074, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.196167] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.196173] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.196360] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:16.196425] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:16.196511] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:16.196531] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:16.196988] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=54] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.198062] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.198061] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.199915] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:16.204057] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=3][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:16.204111] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=44] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:16.204130] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=16][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:16.204142] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:16.204154] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=11][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:16.204163] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=7][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:16.204169] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:16.204176] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:16.204182] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:16.204197] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=14][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:16.204202] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:16.204207] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:16.204212] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:16.204217] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:16.204222] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:16.204202] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=37] PNIO [ratelimit] time: 1713774196204199, bytes: 66195803, bw: 0.024091 MB/s, add_ts: 1001220, add_bytes: 25292 [2024-04-22 16:23:16.204234] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=6][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:16.204241] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.204247] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.204252] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:16.204257] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:16.204264] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:16.204269] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.204285] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:16.204298] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.204303] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.204307] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:16.204320] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:16.204328] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=7][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:16.204334] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:16.204341] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:16.204346] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:16.204351] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd227a050, start=1713774196203932, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:16.204363] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:16.204368] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=4][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:16.204417] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=6][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:16.204423] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED83-0-0] [lt=5][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:16.205136] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.205180] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=7000, remain_us=1951052, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.209700] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:16.209809] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=75] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:16.209843] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=33] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:23:16.212745] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=61] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:16.214182] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=16] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:16.216030] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=147][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:16.216095] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.216107] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=57] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:16.216173] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=60][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:16.216176] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=8000, remain_us=1940056, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.216202] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=27][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:16.216228] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=24][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:16.216252] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=17][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:16.216270] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=17][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:16.216294] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=15][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:16.216312] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=16][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:16.216322] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:16.216332] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=9][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:16.216342] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:16.216352] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:16.216362] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:16.216384] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=21][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:16.216406] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:16.216417] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.216431] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.216479] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=46][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:16.216490] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:16.216503] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:16.216514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.216545] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=27][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:16.216593] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=40][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.216611] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.216637] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=24][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:16.216663] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=13][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:16.216700] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=34][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:16.216725] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=22][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:16.216743] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=16][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:16.216769] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=23][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:16.216789] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=17][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45cc050, start=1713774196215817, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:16.216813] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=21][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:16.216826] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:16.216911] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=14][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:16.216900] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=44] PNIO [ratelimit] time: 1713774196216897, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007642, add_bytes: 0 [2024-04-22 16:23:16.216944] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=30][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:16.216979] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=33][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:16.216994] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=14][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:16.217007] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=9][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:16.217021] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=11][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:16.217031] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED7F-0-0] [lt=10][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:16.226883] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFDCF-0-0] [lt=115][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:16.228052] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.228131] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=1928101, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.229210] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:16.241057] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.241166] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=10000, remain_us=1915067, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.242544] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3591-0-0] [lt=247][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:16.242609] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3591-0-0] [lt=62][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:16.242659] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3591-0-0] [lt=47][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:16.242774] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3591-0-0] [lt=112][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:16.251082] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=60] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:16.251322] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=42][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:16.253891] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.253990] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=11000, remain_us=1902242, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.255304] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.255387] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=86000, remain_us=18571755, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=86, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.268127] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.268207] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=12000, remain_us=1888026, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.284330] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.284413] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=1871819, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.301187] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.301271] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=14000, remain_us=1854961, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.301349] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301389] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301407] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301495] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=85] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301511] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301524] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301566] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301582] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301596] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301609] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301623] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301636] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301650] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301663] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301676] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.301689] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:16.302307] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:16.302375] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197510469743, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774196302294) [2024-04-22 16:23:16.302413] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774196102169, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:16.319340] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.319426] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=1836806, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.338270] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.338417] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=16000, remain_us=1817816, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.345617] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.345700] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=87000, remain_us=18481443, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=87, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.351375] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C24-0-0] [lt=100] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=0, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:16.351533] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C24-0-0] [lt=122] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=13, avg_log_batch_cnt=0, total_group_log_size=1583, avg_group_log_size=121) [2024-04-22 16:23:16.351577] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C24-0-0] [lt=38] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=1629) [2024-04-22 16:23:16.357474] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT WRITE LOG COST TIME](cur_stat_count=3, stat_interval=1000000, avg (count/cost)=5582, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:16.357537] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=1583) [2024-04-22 16:23:16.357594] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=4322, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:16.357719] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=61] [PALF STAT FLUSH CB COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=70, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:16.358663] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.358761] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=17000, remain_us=1797471, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.358903] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF25C-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=14, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:16.358951] INFO [PALF] read (log_cache.cpp:100) [10869][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1583, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:16.358997] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF25C-0-0] [lt=91] [PALF STAT LOG LIFE TIME](cur_stat_count=13, stat_interval=1000000, avg (count/cost)=82351, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:16.359024] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF25C-0-0] [lt=24] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=13, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=20, avg_log_submit_to_flush_time=10196, avg_log_submit_to_slide_time=82330) [2024-04-22 16:23:16.359129] INFO [LIB] stat (utility.h:1165) [10871][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=18] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=40, stat_interval=1000000, avg (count/cost)=397, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:16.359181] INFO [LIB] stat (utility.h:1165) [10871][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=50] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=41, stat_interval=1000000, avg (count/cost)=93, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:16.379663] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.379756] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=18000, remain_us=1776476, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.402003] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.402149] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=19000, remain_us=1754083, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.405236] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:520, tid:10796}]) [2024-04-22 16:23:16.413020] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:16.421413] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:19, local_retry_times:19, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.421619] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=171][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.421658] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.421682] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.421729] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.425271] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=62][errcode=-4725] [2024-04-22 16:23:16.425344] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=70][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.425403] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=55][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.425490] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=83][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.425506] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.425518] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.425563] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=43][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.425612] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.425652] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.425665] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.425683] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=20, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.425717] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=20000, remain_us=1730516, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.425710] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFC75-0-0] [lt=74][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774196425581}) [2024-04-22 16:23:16.429935] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=82] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:16.432839] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:87, local_retry_times:87, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.432933] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.432961] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.432975] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.433059] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.438142] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [2024-04-22 16:23:16.438206] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.438229] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.438244] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.438258] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.438269] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.438280] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.438331] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.438384] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.438406] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:16.438426] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=88, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.438513] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=69] will sleep(sleep_us=88000, remain_us=18388630, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=88, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.445848] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=49][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:20, local_retry_times:20, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.445927] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.445945] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.445954] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.445979] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.449033] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] [2024-04-22 16:23:16.449071] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=36][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.449098] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.449117] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.449132] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.449139] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.449147] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.449192] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.449230] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.449243] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.449259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=21, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.449288] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16] will sleep(sleep_us=21000, remain_us=1706944, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.470500] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:21, local_retry_times:21, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.470611] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=82][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.470649] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.470670] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.470707] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.474292] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] [2024-04-22 16:23:16.474319] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.474332] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.474342] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.474353] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.474361] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.474367] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.474392] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.474409] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.474413] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=3][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.474419] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=22, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.474448] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5] will sleep(sleep_us=22000, remain_us=1681784, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.496567] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:22, local_retry_times:22, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.496657] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.496679] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.496690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.496739] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.498753] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] [2024-04-22 16:23:16.498799] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.498828] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.498845] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.498860] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.498871] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.498882] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.498950] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.498990] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.499003] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.499020] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=23, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.499055] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=23000, remain_us=1657177, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.502492] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=120][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:16.502551] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:16.502588] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774196502480) [2024-04-22 16:23:16.502635] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774196302534, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:16.522242] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:23, local_retry_times:23, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.522339] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=69][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.522379] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=35][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.522399] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.522478] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.525058] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=33][errcode=-4725] [2024-04-22 16:23:16.525120] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=59][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.525163] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.525188] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.525208] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.525225] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.525242] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.525307] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.525363] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.525381] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.525406] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=24, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.525489] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26] will sleep(sleep_us=24000, remain_us=1630744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.526768] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=88][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:88, local_retry_times:88, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.526915] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=112][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.526944] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.526962] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.526991] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.529098] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:16.529140] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.529170] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.529192] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.529250] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=56][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.529264] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.529275] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.529347] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.529384] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.529397] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:16.529414] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=89, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.529478] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17] will sleep(sleep_us=89000, remain_us=18297665, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=89, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.549679] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=51][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:24, local_retry_times:24, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.549804] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=96][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.549834] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.549851] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.549914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.553903] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] [2024-04-22 16:23:16.553969] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=63][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.554011] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.554037] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.554058] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.554075] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.554091] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.554184] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=41] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.554238] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.554259] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.554284] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=25, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.554331] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27] will sleep(sleep_us=25000, remain_us=1601902, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.579582] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:25, local_retry_times:25, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.579681] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=70][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.579720] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.579740] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.579777] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.583763] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=52][errcode=-4725] [2024-04-22 16:23:16.583807] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.583836] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.583852] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.583866] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.583878] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.583888] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.583936] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.583975] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.583988] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.584005] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=26, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.584040] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=26000, remain_us=1572192, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.610232] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:26, local_retry_times:26, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.610321] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.610343] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.610354] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.610380] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.613203] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:16.613249] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.613280] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.613298] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.613312] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.613324] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.613396] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=70][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.613484] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.613542] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.613564] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.613584] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=27, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.613620] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19] will sleep(sleep_us=27000, remain_us=1542613, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.618646] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:89, local_retry_times:89, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.618736] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.618757] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.618768] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.618792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.621014] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:16.621075] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=59][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.621106] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.621124] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.621137] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.621148] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.621159] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.621208] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.621246] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.621259] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:16.621276] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=90, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.621310] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=90000, remain_us=18205832, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=90, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.624392] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=80] Cache replace map node details(ret=0, replace_node_count=0, replace_time=11141, replace_start_pos=943710, replace_num=62914) [2024-04-22 16:23:16.624422] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:16.630425] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=72] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:16.640786] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:27, local_retry_times:27, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.640872] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.640894] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.640905] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.640951] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.644588] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] [2024-04-22 16:23:16.644640] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=49][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.644666] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.644682] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.644696] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.644707] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.644718] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.644765] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.644803] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.644816] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.644833] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=28, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.644890] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39] will sleep(sleep_us=28000, remain_us=1511342, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.673035] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:28, local_retry_times:28, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.673110] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=55][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.673133] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.673144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.673165] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.676936] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] [2024-04-22 16:23:16.676978] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.677009] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.677032] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.677056] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.677074] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.677115] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.677187] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.677246] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.677268] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.677288] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=29, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.677323] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=29000, remain_us=1478910, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.702693] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:16.702801] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774196702680) [2024-04-22 16:23:16.702849] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774196502655, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:16.702899] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:16.706489] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:29, local_retry_times:29, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.706577] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.706599] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.706610] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.706658] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.709553] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=49] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:16.710288] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.710341] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [10461][pnio1][T0][YB42AC175881-000616AB0B5FEDCF-0-0] [lt=49] ts response task statistics(alloc_count=3630, free_count=1814) [2024-04-22 16:23:16.710372] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=30000, remain_us=1445860, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.715175] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.715261] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=91000, remain_us=18111881, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=91, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.738207] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3594-0-0] [lt=357][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:16.738254] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3594-0-0] [lt=45][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:16.738291] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3594-0-0] [lt=35][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:16.738309] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3594-0-0] [lt=17][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:16.744679] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.744780] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=31000, remain_us=1411453, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.379663] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.379756] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=18000, remain_us=1776476, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.402003] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.402149] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=19000, remain_us=1754083, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.405236] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:520, tid:10796}]) [2024-04-22 16:23:16.413020] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:16.421413] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:19, local_retry_times:19, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.421619] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=171][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.421658] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.421682] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.421729] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.425271] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=62][errcode=-4725] [2024-04-22 16:23:16.425344] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=70][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.425403] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=55][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.425490] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=83][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.425506] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.425518] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.425563] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=43][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.425612] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.425652] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.425665] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.425683] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=20, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.425717] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=20000, remain_us=1730516, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.425710] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFC75-0-0] [lt=74][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774196425581}) [2024-04-22 16:23:16.429935] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=82] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:16.432839] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:87, local_retry_times:87, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.432933] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.432961] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.432975] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.433059] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.438142] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [2024-04-22 16:23:16.438206] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.438229] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.438244] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.438258] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.438269] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.438280] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.438331] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.438384] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.438406] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:16.438426] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=88, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.438513] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=69] will sleep(sleep_us=88000, remain_us=18388630, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=88, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.445848] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=49][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:20, local_retry_times:20, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.445927] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.445945] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.445954] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.445979] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.449033] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] [2024-04-22 16:23:16.449071] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=36][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.449098] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.449117] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.449132] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.449139] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.449147] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.449192] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.449230] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.449243] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.449259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=21, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.449288] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16] will sleep(sleep_us=21000, remain_us=1706944, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.470500] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:21, local_retry_times:21, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.470611] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=82][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.470649] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.470670] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.470707] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.474292] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] [2024-04-22 16:23:16.474319] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.474332] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.474342] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.474353] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.474361] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.474367] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.474392] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.474409] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.474413] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=3][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.474419] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=22, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.474448] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=5] will sleep(sleep_us=22000, remain_us=1681784, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.496567] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:22, local_retry_times:22, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.496657] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.496679] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.496690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.496739] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.498753] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] [2024-04-22 16:23:16.498799] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.498828] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.498845] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.498860] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.498871] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.498882] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.498950] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.498990] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.499003] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.499020] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=23, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.499055] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=23000, remain_us=1657177, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.502492] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=120][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:16.502551] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:16.502588] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774196502480) [2024-04-22 16:23:16.502635] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774196302534, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:16.522242] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:23, local_retry_times:23, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.522339] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=69][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.522379] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=35][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.522399] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.522478] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.525058] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=33][errcode=-4725] [2024-04-22 16:23:16.525120] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=59][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.525163] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.525188] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.525208] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.525225] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.525242] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.525307] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.525363] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.525381] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.525406] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=24, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.525489] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26] will sleep(sleep_us=24000, remain_us=1630744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.526768] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=88][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:88, local_retry_times:88, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.526915] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=112][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.526944] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.526962] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.526991] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.529098] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:16.529140] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.529170] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.529192] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.529250] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=56][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.529264] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.529275] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.529347] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.529384] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.529397] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:16.529414] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=89, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.529478] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17] will sleep(sleep_us=89000, remain_us=18297665, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=89, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.549679] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=51][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:24, local_retry_times:24, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.549804] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=96][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.549834] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.549851] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.549914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.553903] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] [2024-04-22 16:23:16.553969] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=63][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.554011] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.554037] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.554058] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.554075] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.554091] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.554184] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=41] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.554238] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.554259] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.554284] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=25, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.554331] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27] will sleep(sleep_us=25000, remain_us=1601902, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.579582] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:25, local_retry_times:25, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.579681] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=70][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.579720] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.579740] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.579777] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.583763] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=52][errcode=-4725] [2024-04-22 16:23:16.583807] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.583836] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.583852] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.583866] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.583878] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.583888] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.583936] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.583975] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.583988] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.584005] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=26, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.584040] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=26000, remain_us=1572192, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.610232] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:26, local_retry_times:26, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.610321] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.610343] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.610354] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.610380] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.613203] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:16.613249] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.613280] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.613298] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.613312] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.613324] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.613396] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=70][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.613484] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.613542] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.613564] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.613584] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=27, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.613620] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19] will sleep(sleep_us=27000, remain_us=1542613, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.618646] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:89, local_retry_times:89, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.618736] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.618757] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.618768] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.618792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.621014] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:16.621075] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=59][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.621106] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.621124] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.621137] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.621148] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.621159] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.621208] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.621246] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.621259] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:16.621276] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=90, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.621310] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=90000, remain_us=18205832, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=90, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.624392] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=80] Cache replace map node details(ret=0, replace_node_count=0, replace_time=11141, replace_start_pos=943710, replace_num=62914) [2024-04-22 16:23:16.624422] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:16.630425] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=72] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:16.640786] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:27, local_retry_times:27, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.640872] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.640894] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.640905] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.640951] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.644588] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=34][errcode=-4725] [2024-04-22 16:23:16.644640] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=49][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.644666] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.644682] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.644696] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.644707] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.644718] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.644765] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.644803] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.644816] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.644833] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=28, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.644890] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39] will sleep(sleep_us=28000, remain_us=1511342, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.673035] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:28, local_retry_times:28, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.673110] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=55][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.673133] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.673144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.673165] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.676936] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] [2024-04-22 16:23:16.676978] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:16.677009] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:16.677032] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:16.677056] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=23][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:16.677074] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:16.677115] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.677187] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.677246] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:16.677268] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:16.677288] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=29, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.677323] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=29000, remain_us=1478910, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.702693] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:16.702801] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774196702680) [2024-04-22 16:23:16.702849] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774196502655, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:16.702899] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:16.706489] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:29, local_retry_times:29, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:16.706577] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.706599] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:16.706610] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:16.706658] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:16.709553] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=49] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:16.710288] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.710341] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [10461][pnio1][T0][YB42AC175881-000616AB0B5FEDCF-0-0] [lt=49] ts response task statistics(alloc_count=3630, free_count=1814) [2024-04-22 16:23:16.710372] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=30000, remain_us=1445860, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.715175] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.715261] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=91000, remain_us=18111881, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=91, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.738207] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3594-0-0] [lt=357][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:16.738254] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3594-0-0] [lt=45][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:16.738291] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3594-0-0] [lt=35][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:16.738309] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3594-0-0] [lt=17][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:16.744679] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.744780] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=31000, remain_us=1411453, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.780068] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.780163] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=32000, remain_us=1376069, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.782502] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:16.782595] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=70] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:16.782628] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:16.782648] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:16.782667] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:16.782686] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:16.782706] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=18][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:16.782745] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=27][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:16.782789] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=40][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:16.782891] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=98][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:16.782924] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=30][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:16.782945] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=19][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:16.782963] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:16.782996] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=19][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:16.783029] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=28][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.783047] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=15][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.783059] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:16.783073] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:16.783097] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=20][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:16.783121] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=21][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.783199] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=32][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:16.783246] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=38][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.783267] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.783278] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=11][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:16.783316] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC73-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:16.783336] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC73-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:16.783353] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:16.783407] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:16.783467] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=56][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:16.783502] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:16.783521] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3ceeb84050, start=1713774196780128, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:16.783598] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196780078), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:23:01.741419, ballot number:9', 'HAPPENED_TIME', '2024-04-22 16:23:01.760528', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:16.786018] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=44][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:16.786086] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:16.786116] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:16.786135] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:16.786152] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:16.786170] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:16.786182] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:16.786198] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:16.786210] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:16.786221] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=10][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:16.786233] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=11][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:16.786243] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=9][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:16.786255] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:16.786275] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:16.786288] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.786302] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.786313] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:16.786325] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:16.786339] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:16.786350] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.786374] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:16.786398] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.786409] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.786418] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:16.786475] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC74-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:16.786497] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC74-0-0] [lt=19][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:16.786513] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:16.786550] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:16.786563] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:16.786578] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:16.786593] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd45f0050, start=1713774196783748, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:16.786671] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196783712), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:23:01.760709', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:16.787920] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=20][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:16.787962] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=31] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:16.787987] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=22][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:16.788006] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:16.788022] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:16.788039] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:16.788051] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:16.788066] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:16.788076] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:16.788086] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=9][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:16.788097] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=10][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:16.788107] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=9][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:16.788118] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:16.788136] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:16.788147] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.788160] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.788171] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:16.788179] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:16.788188] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:16.788195] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.788209] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=12][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:16.788225] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=12][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.788231] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.788237] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:16.788263] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC75-0-0] [lt=6][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:16.788275] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC75-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:16.788285] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:16.788307] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:16.788315] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:16.788324] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:16.788333] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd1498050, start=1713774196786760, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:16.788384] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774196786737), 'ELECTION', 'decentralized to be leader', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'server|lease:["172.23.88.129:2882"|2024-04-22 16:23:05.763,"172.23.88.131:2882"|2024-04-22 16:23:05.763]', 'HAPPENED_TIME', '2024-04-22 16:23:01.764535', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:16.810694] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.810781] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=92000, remain_us=18016362, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=92, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.815762] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.815827] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=33000, remain_us=1340405, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.824566] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:16.830910] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:16.852583] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.852694] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1303539, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.892180] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.892271] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1263961, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.902911] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:16.902970] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=71][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:16.902980] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774196902899) [2024-04-22 16:23:16.903011] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774196702887, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:16.903026] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=37] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:16.903058] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:16.903101] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=40][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:16.903120] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:16.903138] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:16.903151] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:16.903168] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:23:16.903181] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:16.903192] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:16.903204] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:16.903215] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:16.903227] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:16.903239] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:16.903250] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:16.903273] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:16.903296] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=20][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.903310] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:16.903323] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:16.903334] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882, ret=-5019) [2024-04-22 16:23:16.903348] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:16.903360] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:16.903384] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:16.903410] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.903421] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:16.903431] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:16.903505] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=47][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:16.903570] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC13-0-0] [lt=63][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-04-22 16:23:16.903586] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:16.903601] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:16.903614] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:16.903627] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cbe430050, start=1713774196902773, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:16.903645] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:16.903658] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f3ce8856298, table=__all_server, condition=where svr_ip='172.23.88.129' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882, columns_str="zone") [2024-04-22 16:23:16.903686] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:16.903787] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:635) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f3ce8856298, where_condition="where svr_ip='172.23.88.129' and svr_port=2882", zone_name_holder=) [2024-04-22 16:23:16.903832] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:568) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:23:16.903858] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:577) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:23:16.903891] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:16.907792] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.907858] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=93000, remain_us=17919284, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=93, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:16.932796] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.932863] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=36000, remain_us=1223369, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:16.972259] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:16.972335] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=37000, remain_us=1183898, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.000994] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:17.006563] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.006670] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=94000, remain_us=17820472, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=94, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.013389] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.013524] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=38000, remain_us=1142708, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.024793] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:17.031610] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=46] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:17.054789] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.054879] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=39000, remain_us=1101354, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.096108] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.096227] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=1060006, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.101285] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:17.103113] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:17.103268] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=150][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774197103094) [2024-04-22 16:23:17.103377] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=106][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774196903031, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:17.104684] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.104768] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=95000, remain_us=17722375, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=95, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.139646] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.139755] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=41000, remain_us=1016484, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.184804] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.184912] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=42000, remain_us=971320, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.190519] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.190664] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.190672] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:469) [10838][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=7] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-04-22 16:23:17.190719] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=53] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.190728] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [10838][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=52] XA scheduler heartbeat task statistics(avg_time=60) [2024-04-22 16:23:17.190887] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.190921] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.191507] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.191554] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.191648] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=43] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.191718] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.191714] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC11-0-0] [lt=38][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-04-22 16:23:17.191770] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC11-0-0] [lt=53][errcode=-4002] refresh sever list failed(ret=-4002) [2024-04-22 16:23:17.191789] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC11-0-0] [lt=18][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-04-22 16:23:17.192072] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=102] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.192176] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=57] gc stale ls task succ [2024-04-22 16:23:17.192522] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.192676] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.192886] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10609, mem_used=79030272) [2024-04-22 16:23:17.192937] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=25] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:17.193019] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.193315] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=53] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:17.193496] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.193800] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=43] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.193874] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.194080] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:131) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] ====== checkpoint timer task ====== [2024-04-22 16:23:17.194113] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=9] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-22 16:23:17.194157] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=27] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=99) [2024-04-22 16:23:17.194293] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:306) [10844][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] ====== traversal_flush timer task ====== [2024-04-22 16:23:17.194098] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0C-0-0] [lt=29] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:17.194321] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=7] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-22 16:23:17.194381] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=51] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-22 16:23:17.194403] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=17] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=99) [2024-04-22 16:23:17.194518] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=61] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.194462] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0C-0-0] [lt=271] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774193200889, replica_locations:[{server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:8}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774197194424, replica_locations:[{server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:8}, {server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}) [2024-04-22 16:23:17.194483] INFO [MDS] flush (mds_table_mgr.cpp:172) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC0F-0-0] [lt=43] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774207210958129, latest_log_ts:{val:1713774197210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:17.194653] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:338) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC0F-0-0] [lt=160] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-22 16:23:17.194687] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.195235] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1918) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=9] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-22 16:23:17.195286] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1459) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=45] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:17.195301] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=38] current monitor number(seq_=-1) [2024-04-22 16:23:17.195317] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1476) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=27] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:17.195339] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2682) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=19] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:17.195516] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.195665] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-22 16:23:17.196295] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.196959] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:17.197037] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:17.197070] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:17.197099] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:17.197152] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.197616] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.197628] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=60][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:17.197698] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-04-22 16:23:17.197728] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-04-22 16:23:17.197694] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1426) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=55] succ to get rec scn(*this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:17.197792] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=61][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-04-22 16:23:17.197815] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=22][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-04-22 16:23:17.197793] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=96] tx ctx memtable get rec scn(this={ObITable:{this:0x7f3cd10be080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f3cd10be080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:17.197834] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:17.197847] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:17.197865] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-04-22 16:23:17.197849] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:856) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=47] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-22 16:23:17.197878] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:17.197889] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:17.197900] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:17.197911] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:17.197936] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=23][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:17.197949] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:17.197968] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=18][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:17.197942] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774207210958129, latest_log_ts:{val:1713774197210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:17.198071] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=86][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:17.198083] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:599) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=134] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1}) [2024-04-22 16:23:17.198089] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=15][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:17.198142] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=49][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:17.198137] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:470) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=42] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2024-04-22 16:23:17.198156] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:17.198162] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] get rec log scn(service_type_=0, rec_log_ts={val:1713774197210958129, v:0}) [2024-04-22 16:23:17.198168] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:17.198176] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-22 16:23:17.198184] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:17.198190] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=2, rec_log_ts={val:1713773717210958135, v:0}) [2024-04-22 16:23:17.198178] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.198196] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.198214] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:164) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713773717210958135, v:0}, checkpoint_scn_in_ls_meta={val:1713773717210958135, v:0}, ls_id={id:1}, service_type="DAS_ID") [2024-04-22 16:23:17.198223] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=22][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:17.198244] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1579) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-22 16:23:17.198263] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC10-0-0] [lt=33][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:17.198270] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:185) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-22 16:23:17.198288] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:192) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-22 16:23:17.199056] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:17.199809] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC44-0-0] [lt=19][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:17.199880] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC44-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:17.199987] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC44-0-0] [lt=101][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:17.202489] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [10827][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=176] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-04-22 16:23:17.202788] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.202807] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=42] ====== tenant freeze timer task ====== [2024-04-22 16:23:17.202824] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=96000, remain_us=17624318, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=96, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.203185] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713774197617369862, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=-414193, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1713774197203176, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-22 16:23:17.203525] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2401) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=12] Dumping All Cache Objs(alloc_obj_list.count()=6, alloc_obj_list=[{obj_id:99, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175160}, {obj_id:100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174200}, {obj_id:101, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173640}, {obj_id:102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173992}, {obj_id:103, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:176531}, {obj_id:104, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:175513}]) [2024-04-22 16:23:17.203587] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2690) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=58] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:17.205613] INFO pktc_sk_new (pktc_sk_factory.h:78) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO sk_new: s=0x7f3cd332d358 [2024-04-22 16:23:17.205789] INFO pktc_do_connect (pktc_post.h:19) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=28] PNIO sk_new: sk=0x7f3cd332d358, fd=237 [2024-04-22 16:23:17.205842] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] [ussl] write client fd succ, fd:237, gid:0x100000000, need_send_negotiation:1 [2024-04-22 16:23:17.205854] INFO eloop_regist (eloop.c:47) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO sock regist: 0x7f3cd332d358 fd=237 [2024-04-22 16:23:17.205901] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO [ratelimit] time: 1713774197205900, bytes: 66226807, bw: 0.029518 MB/s, add_ts: 1001701, add_bytes: 31004 [2024-04-22 16:23:17.205919] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sock not ready: 0x7f3cd332d358, fd=237 [2024-04-22 16:23:17.205944] INFO ussl_eloop_regist (ussl_eloop.c:39) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x7f3cfd8074d0 fd=237 [2024-04-22 16:23:17.206137] INFO handle_client_writable_event (handle-event.c:157) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=63] [ussl] client send negotiation message succ, fd:237, addr:172.23.88.129:59746, auth_method:NONE, gid:0x100000000 [2024-04-22 16:23:17.206255] INFO clientfd_sk_delete (ussl_factory.c:52) [10457][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=45] [ussl] give back fd to origin epoll succ, client_fd:237, client_epfd:103, event:0x8000000d, client_addr:172.23.88.131:2882, has_error:0 [2024-04-22 16:23:17.206297] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO sock connect OK: 0x7f3cd332d358 fd:237:local:172.23.88.129:59746:remote:172.23.88.131:2882 [2024-04-22 16:23:17.206611] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED84-0-0] [lt=4][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:17.206690] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED84-0-0] [lt=58] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:17.206740] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED84-0-0] [lt=44][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:17.210931] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=65] gts statistics(tenant_id=1, gts_rpc_cnt=8, get_gts_cache_cnt=152, get_gts_with_stc_cnt=4, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-22 16:23:17.218370] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=9] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:17.219353] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=102][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:17.219403] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=40] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:17.219422] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=17][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:17.219454] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=31][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:17.219466] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=11][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:17.219475] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:17.219482] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:17.219490] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:17.219497] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=6][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:17.219503] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:17.219509] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:17.219514] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:17.219520] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED80-0-0] [lt=6][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:17.224449] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=42] PNIO [ratelimit] time: 1713774197224447, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007550, add_bytes: 0 [2024-04-22 16:23:17.224972] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=68] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:17.230600] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.230685] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=43000, remain_us=925548, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.232297] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFE32-0-0] [lt=60][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:17.232309] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=90] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:17.233476] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3597-0-0] [lt=88][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:17.233538] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3597-0-0] [lt=58][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:17.233625] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3597-0-0] [lt=82][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:17.233682] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C3597-0-0] [lt=54][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:17.266858] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10855][T1_MediumLoop][T1][YB42AC175881-000616AB0ADFEBCA-0-0] [lt=15][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:17.266939] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10855][T1_MediumLoop][T1][YB42AC175881-000616AB0ADFEBCA-0-0] [lt=60] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-04-22 16:23:17.266989] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10855][T1_MediumLoop][T1][YB42AC175881-000616AB0ADFEBCA-0-0] [lt=45][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:17.267234] WDIAG [STORAGE.COMPACTION] check_medium_finish_schedule (ob_tenant_medium_checker.cpp:225) [10855][T1_MediumLoop][T1][YB42AC175881-000616AB0ADFEBCA-0-0] [lt=1][errcode=0] failed to refresh ls locality(tmp_ret=-5019) [2024-04-22 16:23:17.267305] INFO [STORAGE] ADD_SUSPECT_INFO (ob_compaction_diagnose.h:828) [10855][T1_MediumLoop][T1][YB42AC175881-000616AB0ADFEBCA-0-0] [lt=31] success to add suspect info(ret=0, info={tenant_id:1, merge_type:"MEDIUM_MERGE", ls_id:{id:9223372036854775807}, tablet_id:{id:9223372036854775807}, add_time:1713774197267278, hash:8494776398607773391}, info_type=11, info_type_str="refresh ls locality cache failed", diagnose_type=3) [2024-04-22 16:23:17.267370] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:170) [10855][T1_MediumLoop][T1][YB42AC175881-000616AB0ADFEBCA-0-0] [lt=50] MediumCheckTask(cost_ts=2314) [2024-04-22 16:23:17.273898] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=53][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:17.277790] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.277898] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=44000, remain_us=878334, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.295893] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=51][errcode=-4018] try free send task failed(ret=-4018) [2024-04-22 16:23:17.300604] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=156] tx gc loop thread is running(MTL_ID()=1) [2024-04-22 16:23:17.300624] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] try gc retain ctx [2024-04-22 16:23:17.300643] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:748) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] get_max_applied_scn(scn={val:1713774187421454783, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:7, palf_committed_end_lsn_:{lsn:358576}, palf_committed_end_scn_:{val:1713774187421454783, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713774187421454783, v:0}}) [2024-04-22 16:23:17.300676] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1023) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] get_min_unreplayed_log_info(lsn={lsn:360327}, scn={val:1713774197617369863, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774196358876, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:360327}, next_to_submit_scn_:{val:1713774197617369863, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:360205}, end_lsn:{lsn:360327}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774197617369862, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774197617369862, v:0}, data_checksum:2923900223, flag:1}}, init_mode_version:0, accumulate_checksum:3370806359, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774197617369862, v:0}}}}}) [2024-04-22 16:23:17.301798] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:253) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774207210958129, latest_log_ts:{val:1713774197210958129, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:17.302338] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.302401] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=97000, remain_us=17524740, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=97, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.303282] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303347] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303318] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:17.303383] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303405] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303423] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303484] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=59] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303503] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303428] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774197303301) [2024-04-22 16:23:17.303522] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303531] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=100][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774197103412, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:17.303562] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303581] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303601] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303621] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303639] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303634] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1713774197617369862, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1713774197303568}, version_delta=-313736) [2024-04-22 16:23:17.303658] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303688] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303708] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:17.303876] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=122] LogLoopThread round_cost_time(us)(round_cost_time=35) [2024-04-22 16:23:17.303960] INFO [PALF] check_parent_health (log_config_mgr.cpp:2764) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=60] re_register_parent reason(palf_id=1, self="172.23.88.129:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713774193299233, last_first_register_time_us=1713773712050886, register_time_us=1713774193299233, curr_time_us=1713774197303958) [2024-04-22 16:23:17.304105] INFO [PALF] check_parent_health (log_config_mgr.cpp:2769) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=72] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:17.307897] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=72][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:145, tid:10829}]) [2024-04-22 16:23:17.325524] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.325555] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1] will sleep(sleep_us=45000, remain_us=830677, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.373689] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.373718] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=46000, remain_us=782514, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.402913] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.402946] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=98000, remain_us=17424195, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=98, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.422566] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.422597] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=47000, remain_us=733634, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.425169] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:17.432996] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=50] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:17.448453] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=67] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=15387, clean_start_pos=4152357, clean_num=125829) [2024-04-22 16:23:17.452656] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C26-0-0] [lt=43] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:17.452741] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C26-0-0] [lt=57] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=1, avg_log_batch_cnt=0, total_group_log_size=122, avg_group_log_size=122) [2024-04-22 16:23:17.452781] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C26-0-0] [lt=36] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=122) [2024-04-22 16:23:17.458949] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=6130, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:17.458973] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=122) [2024-04-22 16:23:17.459007] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=92, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:17.459159] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=83] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=77, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:17.471402] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.471452] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=48000, remain_us=684780, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.471428] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFC9E-0-0] [lt=90][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774197471390}) [2024-04-22 16:23:17.502898] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.502989] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=99000, remain_us=17324153, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=99, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.503513] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:17.503571] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:17.503619] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774197503501) [2024-04-22 16:23:17.503742] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=120][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774197303567, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:17.508626] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=55][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:415, tid:10476}]) [2024-04-22 16:23:17.519573] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:48, local_retry_times:48, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.519652] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=59][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.519680] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.519694] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.519718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.521711] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:17.521773] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=60][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.521802] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.521824] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.521865] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.521879] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.521890] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.521934] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.521984] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.522005] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.522024] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=49, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.522056] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=49000, remain_us=634176, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.571207] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.571290] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.571318] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.571367] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.571427] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.573525] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=70][errcode=-4725] [2024-04-22 16:23:17.573557] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=30][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.573581] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.573597] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.573611] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.573622] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.573633] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.573678] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.573729] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=44][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.573778] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=45][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.573818] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=50, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.573852] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20] will sleep(sleep_us=50000, remain_us=582380, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.602160] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:99, local_retry_times:99, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.602247] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.602275] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.602289] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.602314] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.604069] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] [2024-04-22 16:23:17.604112] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.604142] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.604160] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.604173] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.604185] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.604195] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.604242] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.604306] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=56][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.604327] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:17.604354] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=100, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.604464] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63] will sleep(sleep_us=100000, remain_us=17222711, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=100, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.624088] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:50, local_retry_times:50, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.624169] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.624198] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.624211] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.624235] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.625400] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:17.626174] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:17.626214] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.626245] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.626285] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.626306] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.626323] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.626340] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.626400] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.626496] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=88][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.626518] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.626537] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=51, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.626569] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19] will sleep(sleep_us=51000, remain_us=529663, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.649192] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:17.677725] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:51, local_retry_times:51, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.677847] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=98][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.677903] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=53][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.677921] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.677968] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.679924] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:17.679966] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.679996] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.680014] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.680028] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.680039] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.680050] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.680140] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.680192] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.680232] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=36][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.680252] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=52, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.680333] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=66] will sleep(sleep_us=52000, remain_us=475900, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.693788] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [10825][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=54] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-22 16:23:17.703720] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:17.703785] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774197703707) [2024-04-22 16:23:17.703815] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774197503801, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:17.703846] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:17.704643] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:100, local_retry_times:100, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.704733] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.704755] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.704766] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.704789] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.706813] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [2024-04-22 16:23:17.706852] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.706880] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.706896] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.706931] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.706943] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.706953] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.707000] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.707054] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.707075] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:17.707094] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=101, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.707125] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=17120017, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=101, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.712336] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=53] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:17.732479] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=22][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:52, local_retry_times:52, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.732557] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=58][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.732585] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.732604] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.732636] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.734532] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] [2024-04-22 16:23:17.734571] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.734615] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=42][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.734630] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.734643] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.734654] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.734664] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.734707] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.734759] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.734780] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.734799] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=53, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.734832] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19] will sleep(sleep_us=53000, remain_us=421401, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.739131] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C359A-0-0] [lt=269][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:17.739271] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C359A-0-0] [lt=135][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:17.739338] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C359A-0-0] [lt=64][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:17.739502] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C359A-0-0] [lt=159][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:17.765339] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=145] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:17.787984] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:53, local_retry_times:53, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.788086] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=81][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.788136] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=45][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.788177] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=38][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.788218] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.790173] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:17.790216] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.790269] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=47][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.790312] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.790346] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=32][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.790367] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.790393] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=24][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.790474] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.790527] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.790616] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=85][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.790650] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=54, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.790714] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46] will sleep(sleep_us=54000, remain_us=365519, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.807290] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:101, local_retry_times:101, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.807379] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.807419] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.807482] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=60][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.807510] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.809375] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:17.809424] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.809502] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=75][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.809535] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.809567] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.809595] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.809608] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.809688] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.809785] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=87][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.809860] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=71][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:17.809905] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=42][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=102, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.809953] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32] will sleep(sleep_us=100000, remain_us=17017189, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=102, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.825642] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=103] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:17.844926] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=47][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:54, local_retry_times:54, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.845008] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=59][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.845053] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=41][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.845081] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.845120] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.847221] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:17.847262] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.847355] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=90][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.847379] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.847396] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.847407] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.847417] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.847496] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.847533] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=30][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.847546] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.847563] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=55, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.847595] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=55000, remain_us=308637, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.848195] INFO keepalive_check (time_wheel.c:76) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f3cd332d358,dest=-2091378772:2882), old=0, new=3000000 [2024-04-22 16:23:17.850027] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:17.852906] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C28-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=25, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:17.852981] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C28-0-0] [lt=72] [PALF STAT LOG LIFE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=400234, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:17.852966] INFO [PALF] read (log_cache.cpp:100) [10867][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=122, hit_cnt=1, read_cnt=1, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:17.853009] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C28-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=1, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=75, avg_log_submit_to_flush_time=6363, avg_log_submit_to_slide_time=400159) [2024-04-22 16:23:17.853036] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C28-0-0] [lt=25][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2956, log_task={header:{begin_lsn:{lsn:360327}, end_lsn:{lsn:360449}, log_id:2956, min_scn:{val:1713774197724659836, v:0}, max_scn:{val:1713774197724659836, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:360205}, prev_proposal_id:8, committed_end_lsn:{lsn:360327}, data_checksum:667309872, accum_checksum:3258739754, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774197452643, freeze_ts:1713774197452643, submit_ts:1713774197452718, flushed_ts:1713774197459081, gen_to_freeze cost time:0, gen_to_submit cost time:75, submit_to_flush cost time:6363}, fs_cb_begin_ts=1713774197852877, log_life_time=400234) [2024-04-22 16:23:17.853108] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=57] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=67, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:17.853145] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=37] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=10, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:17.902758] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:55, local_retry_times:55, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.902838] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=58][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.902860] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.902872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.902896] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.903925] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:17.904032] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197617369862, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774197903911) [2024-04-22 16:23:17.904075] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774197703835, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:17.904618] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:17.904658] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.904708] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=47][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.904726] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.904740] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.904751] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.904762] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.904806] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.904858] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=44][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.904879] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.904899] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=56, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.904933] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18] will sleep(sleep_us=56000, remain_us=251299, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:17.910118] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:102, local_retry_times:102, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.910204] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=59][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.910253] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.910268] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.910342] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=56][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.912809] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] [2024-04-22 16:23:17.912855] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.912884] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.912907] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.912928] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.912941] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.912952] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.912999] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.913041] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.913074] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:17.913144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=67][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=103, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.913188] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27] will sleep(sleep_us=100000, remain_us=16913955, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=103, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:17.961089] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:56, local_retry_times:56, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:17.961311] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=196][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.961360] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=46][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:17.961391] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=28][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:17.961494] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:17.964006] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=53][errcode=-4725] [2024-04-22 16:23:17.964043] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:17.964065] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:17.964098] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:17.964120] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:17.964149] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=26][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:17.964162] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.964206] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:17.964277] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=61][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:17.964318] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=36][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:17.964368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=45][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=57, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:17.964472] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=52] will sleep(sleep_us=57000, remain_us=191761, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:18.013331] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:103, local_retry_times:103, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.013500] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=134][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.013542] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.013572] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.013615] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.017481] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:18.017568] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.017644] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16809498, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=104, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.025874] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:18.026254] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.026338] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=0] will sleep(sleep_us=58000, remain_us=129894, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713774198156231) [2024-04-22 16:23:18.050827] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=126] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:18.088788] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=1][errcode=-4012] [2024-04-22 16:23:18.088837] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=47][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.088864] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=25][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:23:18.088891] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=24][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.088944] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=50][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:23:18.088959] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:23:18.088970] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:18.088981] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=9][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:18.088993] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=10][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.089011] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.089033] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=20][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:23:18.089065] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=21][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:18.089094] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=27][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:18.089107] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:23:18.089128] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:23:18.089177] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=42] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1931804) [2024-04-22 16:23:18.089210] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=31][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:23:18.089245] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=30][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:18.089263] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=16][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:23:18.089275] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:18.089289] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=12][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:18.089312] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:18.089428] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=29][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:23:18.089502] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=71][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:18.089517] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=14][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:23:18.089530] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=11][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:18.089577] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=43][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:18.089596] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:521) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=17][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-04-22 16:23:18.089612] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:236) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0A-0-0] [lt=13][errcode=-4012] fail to process refresh task(ret=-4012, ret="OB_TIMEOUT", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-04-22 16:23:18.089652] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:599) [10476][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4012] fail to batch process task(ret=-4012) [2024-04-22 16:23:18.089680] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:499) [10476][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4012] fail to batch execute task(ret=-4012, tasks.count()=1) [2024-04-22 16:23:18.089787] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:261) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:18.089830] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:305) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=36] [REFRESH_SCHEMA] begin refresh schema, (begin_time=1713774198089828) [2024-04-22 16:23:18.089851] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2388) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[]) [2024-04-22 16:23:18.094942] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.095036] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] already timeout, do not need sleep(sleep_us=0, remain_us=1994860, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.098124] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.098197] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=1000, remain_us=1991699, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.101220] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.101301] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=2000, remain_us=1988595, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.102508] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=31] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:18.104133] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:18.104364] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=223][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774197943983343, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774198104118) [2024-04-22 16:23:18.104425] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774197904106, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:18.107129] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.107205] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=3000, remain_us=1982691, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.113867] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.113948] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=4000, remain_us=1975948, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.120558] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.120653] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=5000, remain_us=1969243, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.121221] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.121302] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=16705841, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=105, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.128958] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.129040] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=6000, remain_us=1960856, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.138901] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.138982] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=7000, remain_us=1950914, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.150192] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.150292] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=8000, remain_us=1939604, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.162070] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.162152] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=1927744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.173530] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.173683] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=10000, remain_us=1916214, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.187427] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.187547] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=11000, remain_us=1902349, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.190033] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.190690] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=51] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.190747] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.190813] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=55] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.191048] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.191076] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.191649] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.191711] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.191723] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=62] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.192245] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=59] gc stale ls task succ [2024-04-22 16:23:18.192251] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=58] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.192394] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.192383] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.192643] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [10543][Blacklist][T0][Y0-0000000000000000-0-0] [lt=86] blacklist_loop exec finished(cost_time=60, is_enabled=true, send_cnt=0) [2024-04-22 16:23:18.192732] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=64] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.192849] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.193143] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=63] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10653, mem_used=79030272) [2024-04-22 16:23:18.193145] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=36] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:18.193459] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2024-04-22 16:23:18.193500] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=85] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:18.193515] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=45] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:18.193579] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=33] cannot_recycle_log_size statistics(cannot_recycle_log_size=360937, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:18.193938] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=59] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.194002] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=73] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.194137] INFO pn_ratelimit (group.c:589) [10608][IngressService][T0][Y0-0000000000000000-0-0] [lt=16] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:23:18.194393] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.194596] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.194916] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.196465] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=36] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.196416] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=51] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.197241] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.197645] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=1][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:18.197709] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:18.197719] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.197758] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:18.197782] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:18.198612] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=37][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:18.198695] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=57] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:18.198750] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=49][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:23:18.198781] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=28][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:18.198808] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=24][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:23:18.198839] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=22][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:18.198854] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:18.198873] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:23:18.198885] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:18.198897] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:18.198908] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:18.198943] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=33][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:23:18.198954] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=10][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:23:18.198965] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=10][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:23:18.198975] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:18.198987] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:18.198999] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:18.199011] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:18.199036] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:18.199049] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.199064] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.199075] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:18.199087] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:23:18.199111] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=21][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:18.199124] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.199149] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:18.199175] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.199186] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.199183] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:18.199196] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:18.199227] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC51-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:18.199247] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10899][BlackListServic][T0][YB42AC175881-000616AB07BFEC51-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:23:18.199269] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:18.199284] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:18.199296] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:18.199309] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd227a050, start=1713774198198166, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:18.199329] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:18.199341] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:18.199358] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14] start to print blacklist info [2024-04-22 16:23:18.199500] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11] ls blacklist refresh finish(cost_time=2922) [2024-04-22 16:23:18.202211] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.202316] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=12000, remain_us=1887580, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.207615] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1713774198207612, bytes: 66255052, bw: 0.026890 MB/s, add_ts: 1001712, add_bytes: 28245 [2024-04-22 16:23:18.209077] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:18.209151] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:18.209201] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=44][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:18.209234] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=30][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:18.209257] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=21][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:18.209276] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:18.209299] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=21][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:18.209329] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=20][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:18.209359] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=27][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:18.209373] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:18.209571] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=191][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:18.209618] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=44][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:18.209636] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=17][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:18.209665] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=25][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:18.209680] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=13][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:18.209724] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=27][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:18.209797] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=66][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.209846] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=40][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.209869] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=21][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:18.209907] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=34][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:18.209941] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=30][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:18.209971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=26][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.210031] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=54][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:18.210078] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=39][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.210108] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=27][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.210120] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=11][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:18.210156] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED85-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:18.210193] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=31][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:18.210227] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=29][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:18.210259] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=29][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:18.210275] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=14][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:18.210290] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45cc050, start=1713774198208833, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:18.210373] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=80][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:18.210414] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=35][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:18.210585] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=75][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:18.210617] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED85-0-0] [lt=30][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:18.211327] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:18.211411] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=48] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:18.211482] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=69] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:23:18.213919] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=63] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713774197712249], gts:1713774197756346825, latest_srr:[mts=1713774198213843]}) [2024-04-22 16:23:18.214098] INFO [STORAGE.BLKMGR] inspect_bad_block (ob_block_manager.cpp:1841) [10529][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=91] inspect bad block cost time(cost_time=4101, check_count=0, total_used_macro_block_count=4, max_check_count_per_round=1) [2024-04-22 16:23:18.217377] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.217529] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=13000, remain_us=1872367, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.221105] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=0] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:18.223478] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=191][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:18.223551] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:18.223616] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=59][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:18.223664] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=45][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:18.223710] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=42][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:18.223784] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=65][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:18.223816] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=29][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:18.223836] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=13][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:18.223863] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=24][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:18.223877] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:18.223902] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=22][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:18.223915] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=12][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:18.223932] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=15][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:18.223963] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=28][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:18.223983] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=17][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:18.224035] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=39][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:18.224079] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=39][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.224123] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=39][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.224180] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=54][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:18.224202] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=19][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:18.224243] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=35][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:18.224272] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=26][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.224317] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=40][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:18.224361] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=37][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.224389] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=26][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.224401] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=12][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:18.224424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:18.224490] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=64][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:18.224509] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:18.224550] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=38][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:18.224583] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=29][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:18.224614] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=28][errcode=-5019] query failed(ret=-5019, conn=0x7f3ceeb84050, start=1713774198223190, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:18.224649] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=33][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:18.224679] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=26][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:18.224786] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=30][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:18.224813] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=24][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:18.224836] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=21][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:18.224866] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=28][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:18.224904] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=27][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:18.224923] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.224949] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=40][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:18.224989] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED81-0-0] [lt=39][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:18.225006] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16602136, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=106, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.226085] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=64] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:18.232169] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1713774198232165, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007718, add_bytes: 0 [2024-04-22 16:23:18.234105] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C359D-0-0] [lt=397][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:18.234169] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C359D-0-0] [lt=60][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:18.234234] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C359D-0-0] [lt=60][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:18.234331] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C359D-0-0] [lt=93][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:18.235039] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.235132] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1854764, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.235798] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFE95-0-0] [lt=65][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:18.251565] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=72] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:18.252990] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.253080] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=1836817, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.272021] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.272107] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=16000, remain_us=1817789, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.272895] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=97] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:18.288276] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=27][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:18.292113] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.292170] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=17000, remain_us=1797726, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.304352] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=89][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:18.304398] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774198146501507, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774198304340) [2024-04-22 16:23:18.304414] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774198104516, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:18.305110] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305142] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305153] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305161] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305167] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305172] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305285] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=107] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305322] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305336] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305347] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305362] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305378] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305398] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305414] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305476] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.305519] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:18.312829] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.312869] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=18000, remain_us=1777027, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.327658] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.327703] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=16499439, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=107, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.334341] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.334399] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=19000, remain_us=1755497, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.357569] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.357725] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=20000, remain_us=1732172, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.381688] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.381806] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=21000, remain_us=1708091, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.404491] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:18.404643] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=111][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774198249345480, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774198404476) [2024-04-22 16:23:18.404935] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=283][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774198304422, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:18.406789] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.406864] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=22000, remain_us=1683032, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.426279] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:18.431542] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.431628] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=16395514, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=108, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.433705] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.433798] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=23000, remain_us=1656098, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.452293] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:18.453797] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C32-0-0] [lt=60] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:18.453882] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C32-0-0] [lt=58] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=6, avg_log_batch_cnt=0, total_group_log_size=732, avg_group_log_size=122) [2024-04-22 16:23:18.453917] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C32-0-0] [lt=30] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=732) [2024-04-22 16:23:18.458826] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=732) [2024-04-22 16:23:18.460939] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.461026] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=24000, remain_us=1628870, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.488835] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.488902] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFCCE-0-0] [lt=36][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774198488801}) [2024-04-22 16:23:18.488955] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=1600941, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.518194] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.518304] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=26000, remain_us=1571593, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.536917] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.537023] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16290120, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=109, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.547759] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.547866] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=27000, remain_us=1542031, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.554033] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C34-0-0] [lt=18][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2962, log_task={header:{begin_lsn:{lsn:361059}, end_lsn:{lsn:361181}, log_id:2962, min_scn:{val:1713774198432394303, v:0}, max_scn:{val:1713774198432394303, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:360937}, prev_proposal_id:8, committed_end_lsn:{lsn:361059}, data_checksum:667309872, accum_checksum:4059233697, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774198453784, freeze_ts:1713774198453784, submit_ts:1713774198453860, flushed_ts:1713774198458961, gen_to_freeze cost time:0, gen_to_submit cost time:76, submit_to_flush cost time:5101}, fs_cb_begin_ts=1713774198554012, log_life_time=100228) [2024-04-22 16:23:18.560879] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=63] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=5790, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:18.560959] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=58] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=73, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:18.561061] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=90, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:18.578784] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.578875] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=28000, remain_us=1511021, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.604703] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=68][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:18.604761] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:18.604825] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774198249345480, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774198604691) [2024-04-22 16:23:18.604853] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774198405005, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:18.610757] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.610865] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=29000, remain_us=1479031, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.611587] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:550, tid:10476}]) [2024-04-22 16:23:18.626426] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:18.637164] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:109, local_retry_times:109, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.637258] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.637299] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=37][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.637313] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.637339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.640104] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=82][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:29, local_retry_times:29, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.640204] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=76][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.640255] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=46][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.640297] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=39][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.640381] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=39][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.641083] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] [2024-04-22 16:23:18.641137] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=51][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.641198] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=57][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.641243] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.641277] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.641291] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.641302] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.641388] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.641480] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.641515] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:18.641536] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=110, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.641569] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=16185573, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=110, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.644210] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=136][errcode=-4725] [2024-04-22 16:23:18.644251] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.644290] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=36][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.644322] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=28][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.644352] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=27][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.644371] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.644436] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=61][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.644545] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=71] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.644596] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=44][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.644635] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=35][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.644686] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=47][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=30, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.644746] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=44] will sleep(sleep_us=30000, remain_us=1445150, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.652995] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=73] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:18.654193] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C36-0-0] [lt=116][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2963, log_task={header:{begin_lsn:{lsn:361181}, end_lsn:{lsn:361303}, log_id:2963, min_scn:{val:1713774198544147547, v:0}, max_scn:{val:1713774198544147547, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:361059}, prev_proposal_id:8, committed_end_lsn:{lsn:361181}, data_checksum:667309872, accum_checksum:518940016, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774198553962, freeze_ts:1713774198553962, submit_ts:1713774198553975, flushed_ts:1713774198560994, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:7019}, fs_cb_begin_ts=1713774198654165, log_life_time=100203) [2024-04-22 16:23:18.674946] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=36][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:30, local_retry_times:30, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.675043] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=73][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.675106] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=59][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.675147] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.675203] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.679327] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=50][errcode=-4725] [2024-04-22 16:23:18.679367] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.679391] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.679421] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.679472] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.679504] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=30][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.679540] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.679625] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=34] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.679692] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=59][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.679742] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=45][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.679777] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=32][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=31, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.679832] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37] will sleep(sleep_us=31000, remain_us=1410064, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.704840] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:18.710997] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:31, local_retry_times:31, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.711100] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=78][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.711131] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=29][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.711147] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.711180] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.715063] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=27][errcode=-4725] [2024-04-22 16:23:18.715102] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=36][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.715146] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=42][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.715161] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.715175] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.715186] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.715197] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.715244] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.715297] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.715319] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.715338] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=32, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.715372] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18] will sleep(sleep_us=32000, remain_us=1374525, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.715513] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=89] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:18.731300] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=14][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:18.731377] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:18.731410] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:18.731430] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:18.731527] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=92][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:18.731559] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=23][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:18.731587] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=25][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:18.731607] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:18.731642] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=33][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:18.731655] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=11][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:18.731667] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=10][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:18.731678] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:18.731691] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:18.731714] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:18.731728] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.731743] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.731754] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:18.731768] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:18.731782] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:18.731804] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=20][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.731831] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:18.731868] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=30][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.731886] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.731902] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:18.731931] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC76-0-0] [lt=14][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:18.731960] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC76-0-0] [lt=23][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:18.731977] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:18.732032] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:18.732046] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:18.732061] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:18.732076] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd45f0050, start=1713774198729236, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:18.732148] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198729139), 'ELECTION', 'lease expired to revoke', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'leader lease expired, server|lease:[16:22:31.737|16:22:31.737|16:22:31.737]', 'HAPPENED_TIME', '2024-04-22 16:22:33.713289', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:18.734414] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=38][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:18.734509] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=79] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:18.734539] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:18.734563] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=22][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:18.734591] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=26][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:18.734616] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=19][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:18.734636] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=18][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:18.734659] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=14][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:18.734676] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=15][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:18.734694] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=17][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:18.734706] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=11][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:18.734716] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=9][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:18.734745] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=26][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:18.734765] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:18.734778] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.734792] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.734803] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:18.734814] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:18.734828] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:18.734839] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.734862] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:18.734886] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.734905] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:18.734915] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:18.734944] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC77-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:18.734971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC77-0-0] [lt=22][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:18.734990] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:18.735025] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:18.735038] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:18.735061] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:18.735078] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd1498050, start=1713774198732292, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:18.735140] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198732258), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.129:2882", expired time:16:22:31.737806, ballot number:8', 'HAPPENED_TIME', '2024-04-22 16:22:33.714275', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:18.736934] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=28][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:18.736999] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=48] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:18.737034] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=31][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:18.737061] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=24][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:18.737099] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=36][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:18.737118] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:18.737131] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:18.737147] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:18.737160] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:18.737171] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=10][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:18.737182] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=10][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:18.737192] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=9][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:18.737204] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:18.737223] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:18.737235] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.737248] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:18.737269] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=19][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:18.737281] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774198735235), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.131:2882", reason:the only request', 'HAPPENED_TIME', '2024-04-22 16:22:33.714569', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:18.737295] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC78-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:18.740182] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A0-0-0] [lt=381][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:18.740256] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A0-0-0] [lt=69][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:18.740388] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A0-0-0] [lt=126][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:18.740538] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A0-0-0] [lt=145][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:18.741727] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:110, local_retry_times:110, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.741863] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=101][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.741935] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=68][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.741958] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.741992] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.745949] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] [2024-04-22 16:23:18.745976] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.745992] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.746004] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.746015] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.746025] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.746031] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.746058] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.746078] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.746095] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:18.746105] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=111, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.746123] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] will sleep(sleep_us=100000, remain_us=16081019, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=111, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.747502] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=30][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:32, local_retry_times:32, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.747544] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=32][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.747556] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.747565] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.747585] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.751289] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] [2024-04-22 16:23:18.751348] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=56][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.751379] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.751402] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.751424] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.751477] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=50][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.751490] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.751536] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.751626] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=79][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.751651] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.751672] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=33, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.751704] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18] will sleep(sleep_us=33000, remain_us=1338192, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.754306] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C38-0-0] [lt=85][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2964, log_task={header:{begin_lsn:{lsn:361303}, end_lsn:{lsn:361425}, log_id:2964, min_scn:{val:1713774198645095532, v:0}, max_scn:{val:1713774198645095532, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:361181}, prev_proposal_id:8, committed_end_lsn:{lsn:361303}, data_checksum:667309872, accum_checksum:3876083090, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774198654122, freeze_ts:1713774198654122, submit_ts:1713774198654133, flushed_ts:1713774198658330, gen_to_freeze cost time:0, gen_to_submit cost time:11, submit_to_flush cost time:4197}, fs_cb_begin_ts=1713774198754267, log_life_time=100145) [2024-04-22 16:23:18.784847] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:33, local_retry_times:33, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.784933] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.784963] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.784982] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.785038] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=41][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.788938] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] [2024-04-22 16:23:18.788987] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.789019] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.789037] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.789052] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.789063] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.789074] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.789122] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.789175] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.789190] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.789208] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=34, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.789279] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=53] will sleep(sleep_us=34000, remain_us=1300617, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.804946] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:18.805014] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774198544147547, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774198804934) [2024-04-22 16:23:18.805051] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774198604913, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:18.823360] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:34, local_retry_times:34, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.823477] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.823500] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.823512] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.823563] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.827473] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=57][errcode=-4725] [2024-04-22 16:23:18.827518] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=42][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.827550] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.827573] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.827590] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.827623] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=31][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.827634] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.827682] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.827737] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.827764] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.827805] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=35, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.827864] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37] will sleep(sleep_us=35000, remain_us=1262033, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.837936] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=159] Cache replace map node details(ret=0, replace_node_count=0, replace_time=11207, replace_start_pos=1006624, replace_num=62914) [2024-04-22 16:23:18.837989] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:18.846238] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:111, local_retry_times:111, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.846290] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.846305] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.846314] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.846330] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.849992] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] [2024-04-22 16:23:18.850016] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.850031] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.850040] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.850048] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.850054] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.850059] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.850088] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.850108] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.850115] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:18.850124] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=112, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.850143] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] will sleep(sleep_us=100000, remain_us=15976999, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=112, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.853772] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:18.862531] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DBE-0-0] [lt=172] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:18.862590] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DBE-0-0] [lt=57] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=34893, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:18.862579] INFO [PALF] read (log_cache.cpp:100) [10867][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:18.862619] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175883-000616AB09173DBE-0-0] [lt=26] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=5700, avg_log_submit_to_slide_time=34874) [2024-04-22 16:23:18.862731] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=17] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=85, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:18.862776] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=44] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:18.863037] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=52][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:35, local_retry_times:35, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.863139] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=70][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.863162] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.863173] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.863203] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.866388] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=79][errcode=-4725] [2024-04-22 16:23:18.866433] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=42][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.866500] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=64][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.866515] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.866530] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.866542] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.866554] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.866782] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=171] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.866852] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=60][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.866883] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.866906] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=36, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.866941] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19] will sleep(sleep_us=36000, remain_us=1222956, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.903099] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:36, local_retry_times:36, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.903192] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.903215] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.903227] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.903275] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.905823] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=41][errcode=-4725] [2024-04-22 16:23:18.905905] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=79][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.905939] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=31][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.905960] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.905981] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.906007] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.906027] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.906094] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.906160] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=57][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.906177] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.906194] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=37, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.906240] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=28] will sleep(sleep_us=37000, remain_us=1183656, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.943418] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=54][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:37, local_retry_times:37, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.943547] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=107][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.943569] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.943580] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.943605] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.947388] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:18.947469] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.947496] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.947512] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.947526] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.947538] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.947549] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.947619] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=30] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.947676] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.947698] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:18.947723] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=38, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.947761] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22] will sleep(sleep_us=38000, remain_us=1142136, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:18.950317] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:112, local_retry_times:112, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.950411] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.950472] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=58][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.950492] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.950519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.954090] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:18.954140] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:18.954179] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:18.954204] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:18.954246] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:18.954264] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:18.954279] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.954357] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.954415] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=49][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:18.954489] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=69][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:18.954545] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=52][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=113, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:18.954602] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39] will sleep(sleep_us=100000, remain_us=15872541, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=113, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:18.985910] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:38, local_retry_times:38, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:18.986002] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=69][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.986031] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:18.986050] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:18.986099] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:18.990258] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:18.990345] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=39000, remain_us=1099552, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.005137] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:19.005278] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774198849496820, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774199005122) [2024-04-22 16:23:19.005312] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774198805074, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:19.034530] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.034599] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=40000, remain_us=1055297, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.038131] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:19.054499] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=69] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:19.054774] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=43][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:19.059259] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.059564] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15767578, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=114, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.078511] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.078605] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=41000, remain_us=1011292, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.103652] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=68] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:19.104653] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED40-0-0] [lt=48] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:19.111259] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [10835][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=81] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f3cec9dc2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-22 16:23:19.123973] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.124063] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=42000, remain_us=965833, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.163574] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.163673] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=15663470, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=115, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.169089] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.169153] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=43000, remain_us=920743, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.190236] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=67] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.190532] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.190857] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=47] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.190901] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=44] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.190988] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=55] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.191212] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.191404] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.191753] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.191775] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.191866] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.191918] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:222) [10510][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=51] [ASYNC TASK QUEUE](queue_size=0) [2024-04-22 16:23:19.192296] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.192364] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=20] gc stale ls task succ [2024-04-22 16:23:19.192935] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=78] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.193107] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.193192] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.193373] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=42] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:19.193407] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.193382] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=53] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10701, mem_used=79030272) [2024-04-22 16:23:19.193629] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=29] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:19.194062] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=60] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.194561] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.194758] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=90] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.195626] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=35] current monitor number(seq_=-1) [2024-04-22 16:23:19.195749] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.195846] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=71] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.196775] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.197401] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=74] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.197518] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=50] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.198488] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=164][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:19.198567] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=58][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:19.198602] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:19.198634] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:19.198769] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:19.204944] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=26] ====== tenant freeze timer task ====== [2024-04-22 16:23:19.205349] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:19.205400] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774199050619316, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774199205337) [2024-04-22 16:23:19.205474] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774199005334, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:19.208148] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=63] PNIO [ratelimit] time: 1713774199208144, bytes: 66287619, bw: 0.031042 MB/s, add_ts: 1000532, add_bytes: 32567 [2024-04-22 16:23:19.212749] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED86-0-0] [lt=12][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:19.212829] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED86-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:19.212864] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED86-0-0] [lt=30][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:19.213251] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED86-0-0] [lt=0][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:19.213296] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED86-0-0] [lt=42][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:19.213372] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=85][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:47, tid:10899}]) [2024-04-22 16:23:19.216031] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.216150] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=33] will sleep(sleep_us=44000, remain_us=873747, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.226419] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=35] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:19.228795] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=231][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:19.228864] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=49] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:19.228899] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:19.229115] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=210][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:19.229185] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=66][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:19.229229] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=34][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:19.229244] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:19.229261] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:19.229275] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:19.229286] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:19.229298] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:19.229310] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:19.229322] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:19.229335] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:19.229347] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:19.229377] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:19.229392] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:19.229411] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:19.229432] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=19][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:19.229485] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=48][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:19.229502] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:19.229515] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.229542] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=22][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:19.229569] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:19.229581] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:19.229591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:19.229626] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:19.229644] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=17][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:19.229661] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:19.229688] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=25][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:19.229701] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:19.229716] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f3d1177a050, start=1713774199228557, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:19.229772] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=54][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:19.229786] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:19.229904] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=15][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:19.229943] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=35][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:19.229967] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=22][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:19.229984] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=15][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:19.230001] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=11][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:19.230018] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=13][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:19.230041] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED82-0-0] [lt=22][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:19.235609] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A3-0-0] [lt=260][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:19.235676] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A3-0-0] [lt=63][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:19.235764] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A3-0-0] [lt=84][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:19.235802] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A3-0-0] [lt=36][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:19.238306] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:19.239737] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=66] PNIO [ratelimit] time: 1713774199239733, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007568, add_bytes: 0 [2024-04-22 16:23:19.240202] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFEF8-0-0] [lt=63][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:19.255005] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:19.265298] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.265404] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=45000, remain_us=824493, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.267603] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.267651] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15559491, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=116, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.307019] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307091] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=69] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307118] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307141] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307157] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307192] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307208] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307231] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307260] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307304] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307326] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307339] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307353] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307366] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307379] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.307403] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:19.310612] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=81][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:19.314269] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.314370] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=46000, remain_us=775527, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.364162] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.364303] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=47000, remain_us=725593, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.370800] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.370891] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15456251, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=117, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.405626] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:19.405709] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774199248799022, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774199405609) [2024-04-22 16:23:19.405804] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774199205494, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:19.415522] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.415613] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=48000, remain_us=674283, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.438500] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=63] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:19.455098] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C42-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=1, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:19.455208] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C42-0-0] [lt=74] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=8, avg_log_batch_cnt=0, total_group_log_size=967, avg_group_log_size=120) [2024-04-22 16:23:19.455244] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C42-0-0] [lt=31] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=967) [2024-04-22 16:23:19.455690] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:19.460648] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=967) [2024-04-22 16:23:19.462265] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1172) [10870][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=151] succ to submit log task to replay service(to_submit_lsn={lsn:362082}, to_submit_scn={val:1713774199448609235, v:0}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774199462159, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:362148}, next_to_submit_scn_:{val:1713774199448609236, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:362026}, end_lsn:{lsn:362148}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774199448609235, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774199448609235, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:981392176, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774199448609235, v:0}}}}}) [2024-04-22 16:23:19.467827] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.467941] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=49000, remain_us=621956, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.472711] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=73] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=16945, clean_start_pos=4278186, clean_num=125829) [2024-04-22 16:23:19.474134] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.474207] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=15352936, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=118, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.521117] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.521195] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFCF2-0-0] [lt=42][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774199521078}) [2024-04-22 16:23:19.521225] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=50000, remain_us=568671, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.577342] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.577433] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=51000, remain_us=512464, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.579720] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.579808] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=15247335, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=119, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.605866] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:19.605925] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:19.606051] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774199448609235, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774199605854) [2024-04-22 16:23:19.606100] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774199405824, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:19.632546] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.632658] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=52000, remain_us=457239, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.638711] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=63] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:19.660534] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=84] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=5375, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:19.660572] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=87, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:19.660717] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=98, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:19.673202] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:19.684317] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.684419] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15142724, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=120, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.688748] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.688840] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=53000, remain_us=401057, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.706024] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:19.715133] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=103][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:325, tid:10796}]) [2024-04-22 16:23:19.718345] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=50] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:19.719192] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [10464][pnio1][T0][YB42AC175881-000616AB0B5FEDD5-0-0] [lt=31] ts response task statistics(alloc_count=3654, free_count=1826) [2024-04-22 16:23:19.741595] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A6-0-0] [lt=372][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:19.741703] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A6-0-0] [lt=103][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:19.741857] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A6-0-0] [lt=149][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:19.741913] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A6-0-0] [lt=54][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:19.742022] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:53, local_retry_times:53, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:19.742101] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.742159] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=54][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.742175] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:19.742203] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:19.745811] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:19.745855] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:19.745888] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:19.745906] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:19.745921] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:19.745933] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:19.745944] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.745993] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.746048] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.746071] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:19.746114] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=40][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=54, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.746163] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=31] will sleep(sleep_us=54000, remain_us=343733, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.778276] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=148] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:19.784706] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=81][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:120, local_retry_times:120, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:19.784782] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=51][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.784803] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.784817] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:19.784856] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:19.788356] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] [2024-04-22 16:23:19.788400] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=42][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:19.788465] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:19.788478] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:19.788488] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:19.788497] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:19.788505] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.788546] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.788588] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=36][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.788605] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:19.788620] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=121, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.788663] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29] will sleep(sleep_us=100000, remain_us=15038480, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=121, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.800302] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:54, local_retry_times:54, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:19.800380] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=57][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.800408] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.800484] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=73][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:19.800540] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:19.804147] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=28][errcode=-4725] [2024-04-22 16:23:19.804194] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:19.804226] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:19.804270] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=41][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:19.804292] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:19.804309] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:19.804325] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.804425] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.804527] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=94][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.804550] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:19.804570] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=55, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.804602] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18] will sleep(sleep_us=55000, remain_us=285294, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.806160] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:19.806257] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774199651382258, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774199806140) [2024-04-22 16:23:19.806304] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774199606129, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:19.813892] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:19.813969] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:19.814001] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:19.814021] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:19.814081] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=57][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:19.814104] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=17][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:19.814117] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:19.814135] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:19.814147] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:19.814159] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=10][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:19.814170] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=10][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:19.814181] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:19.814234] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:19.814280] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=34][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:19.814303] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=18][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:19.814326] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:19.814345] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=16][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:19.814374] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=25][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:19.814390] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:19.814402] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.814426] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:19.814491] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=57][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:19.814505] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:19.814514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:19.814548] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC79-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:19.814575] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC79-0-0] [lt=22][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:19.814602] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:19.814645] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:19.814658] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:19.814674] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:19.814689] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cc12b0050, start=1713774199811654, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:19.814759] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774199811571), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:32.234907, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:34.805824', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:19.838930] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:19.859797] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:55, local_retry_times:55, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:19.859885] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.859936] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=47][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.859949] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:19.859975] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:19.864091] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:19.864161] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=67][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:19.864201] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=36][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:19.864226] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:19.864242] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:19.864254] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:19.864271] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.864337] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.864411] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=63][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.864542] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=126][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:19.864578] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=32][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=56, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.864618] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22] will sleep(sleep_us=56000, remain_us=225279, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.873812] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:19.888807] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:121, local_retry_times:121, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:19.888899] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.888928] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.888942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:19.888968] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:19.892577] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:19.892633] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=53][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:19.892666] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:19.892695] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:19.892740] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=42][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:19.892754] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:19.892766] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.892820] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.892859] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.892872] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:19.892932] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=57][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=122, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.892982] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32] will sleep(sleep_us=100000, remain_us=14934161, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=122, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:19.920885] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:56, local_retry_times:56, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:19.920971] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.920999] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.921017] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:19.921073] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:19.924992] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22][errcode=-4725] [2024-04-22 16:23:19.925033] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:19.925081] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=44][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:19.925103] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:19.925124] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:19.925141] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:19.925160] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.925213] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.925269] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.925290] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:19.925310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=57, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.925343] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18] will sleep(sleep_us=57000, remain_us=164554, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.982493] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:57, local_retry_times:57, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:19.982606] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=89][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.982628] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.982639] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:19.982664] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:19.988170] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22][errcode=-4725] [2024-04-22 16:23:19.988223] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=50][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:19.988265] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=39][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:19.988282] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:19.988297] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:19.988308] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:19.988318] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.988367] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.988476] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=66][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.988499] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:19.988526] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=58, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.988561] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18] will sleep(sleep_us=58000, remain_us=101335, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713774200089895) [2024-04-22 16:23:19.993191] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:122, local_retry_times:122, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:19.993335] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=92][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.993366] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:19.993710] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=338][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:19.993814] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=86][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:19.999087] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] [2024-04-22 16:23:19.999148] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=58][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:19.999182] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:19.999207] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:19.999230] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:19.999248] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:19.999263] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.999335] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:19.999406] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:19.999428] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:19.999527] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=95][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=123, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:19.999572] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28] will sleep(sleep_us=100000, remain_us=14827571, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=123, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.006407] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:20.006507] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=96][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774199651382258, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774200006394) [2024-04-22 16:23:20.006542] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774199806336, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:20.039129] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:20.046747] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=40][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:58, local_retry_times:58, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.046838] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.046866] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.046885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.046941] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.050968] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4012] [2024-04-22 16:23:20.051014] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=44][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.051064] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=46][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:23:20.051083] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.051097] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:23:20.051108] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:23:20.051118] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:20.051130] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:20.051142] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.051160] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.051182] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:23:20.051213] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:20.051224] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:20.051235] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:23:20.051264] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:23:20.051284] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1959748) [2024-04-22 16:23:20.051300] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:23:20.051320] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:20.051341] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:23:20.051360] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:20.051381] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:20.051412] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=25][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:20.051596] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=93][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:23:20.051649] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=51][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:20.051671] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:23:20.051688] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:20.051707] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:20.051724] WDIAG [SHARE.SCHEMA] operator() (ob_multi_version_schema_service.cpp:2425) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4012] fail to get baseline_schema_version(ret=-4012) [2024-04-22 16:23:20.051738] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2495) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11] [REFRESH_SCHEMA] end refresh and add schema(ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], cost=1961857) [2024-04-22 16:23:20.051757] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:308) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4012] fail to refresh and add schema(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:20.051769] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:314) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11] [REFRESH_SCHEMA] end refresh schema with new mode, (ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], used time=1961940) [2024-04-22 16:23:20.055865] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C48-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=30, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:20.055925] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C48-0-0] [lt=58] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=39281, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:20.055940] INFO [PALF] read (log_cache.cpp:100) [10871][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=723, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:20.055953] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C48-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=37, avg_log_submit_to_flush_time=5630, avg_log_submit_to_slide_time=39244) [2024-04-22 16:23:20.056050] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C48-0-0] [lt=94][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2972, log_task={header:{begin_lsn:{lsn:362270}, end_lsn:{lsn:362392}, log_id:2972, min_scn:{val:1713774199851854328, v:0}, max_scn:{val:1713774199851854328, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:362148}, prev_proposal_id:8, committed_end_lsn:{lsn:362270}, data_checksum:667309872, accum_checksum:2377621667, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774199855511, freeze_ts:1713774199855512, submit_ts:1713774199855524, flushed_ts:1713774199859842, gen_to_freeze cost time:1, gen_to_submit cost time:13, submit_to_flush cost time:4318}, fs_cb_begin_ts=1713774200055835, log_life_time=200324) [2024-04-22 16:23:20.056107] INFO [LIB] stat (utility.h:1165) [10869][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=47] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=143, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:20.056155] INFO [LIB] stat (utility.h:1165) [10869][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=46] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=11, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:20.058637] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] [2024-04-22 16:23:20.058670] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=31][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.058693] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.058710] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.058721] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.058730] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.058738] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.058774] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.058817] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.058828] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.058841] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.058875] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19] already timeout, do not need sleep(sleep_us=0, remain_us=1992920, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.058896] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.058928] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.058941] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.058949] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.058965] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.064656] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] [2024-04-22 16:23:20.064771] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=112][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.064804] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.064828] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.064845] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.064856] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.064866] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.064917] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.064988] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=62][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.065003] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.065021] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.065055] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18] will sleep(sleep_us=1000, remain_us=1986740, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.066269] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=99][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.066338] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=50][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.066361] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.066379] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.066423] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.069105] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=52][errcode=-4725] [2024-04-22 16:23:20.069137] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=31][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.069162] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.069176] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.069187] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.069197] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.069206] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.069261] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=24] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.069306] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=38][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.069325] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.069342] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.069370] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15] will sleep(sleep_us=2000, remain_us=1982426, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.071503] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=24][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.071552] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=35][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.071571] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.071582] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.071601] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.074321] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] [2024-04-22 16:23:20.074383] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=60][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.074412] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.074428] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.074477] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=47][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.074488] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.074498] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.074538] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.074542] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=57] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:20.074586] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.074605] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.074622] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=3, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.074651] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16] will sleep(sleep_us=3000, remain_us=1977144, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.077807] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:3, local_retry_times:3, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.077873] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.077895] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.077913] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.077953] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.081555] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] [2024-04-22 16:23:20.081603] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.081634] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.081652] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.081666] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.081677] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.081687] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.081838] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=113] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.081887] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.081903] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.081922] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=4, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.081954] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18] will sleep(sleep_us=4000, remain_us=1969841, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.086114] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:4, local_retry_times:4, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.086201] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=68][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.086223] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.086234] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.086257] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.089914] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.090004] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=5000, remain_us=1961791, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.098954] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.099051] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=6000, remain_us=1952744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.099735] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:20.103037] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.103130] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14724013, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=124, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.104847] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=35] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:20.108890] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.108972] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=7000, remain_us=1942823, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.119691] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.119743] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1932052, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.131119] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.131182] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=9000, remain_us=1920613, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.143852] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.143935] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=10000, remain_us=1907861, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.157934] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.158015] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=11000, remain_us=1893781, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.173018] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.173121] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=12000, remain_us=1878675, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.188694] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.188792] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=1863004, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.190269] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.190392] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=56] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.190982] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=49] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.191076] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=53] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.191147] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=50] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.191256] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.191585] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=77] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.191699] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=56] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.191862] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.191872] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.192489] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=23] gc stale ls task succ [2024-04-22 16:23:20.192645] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.193256] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.193475] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.193471] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.193577] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=55] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:20.193608] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=110] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10743, mem_used=79030272) [2024-04-22 16:23:20.193784] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=73] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:20.193770] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=26] ====== check clog disk timer task ====== [2024-04-22 16:23:20.193782] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=67] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.193818] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=37] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:20.193867] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=24] cannot_recycle_log_size statistics(cannot_recycle_log_size=362636, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:20.194269] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=32] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.194331] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.194867] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.195000] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.195281] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:20.195317] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:20.195334] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=14][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:20.195345] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:20.195355] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=9][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:20.195364] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:20.195369] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:20.195383] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:20.195389] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:20.195422] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=17][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:20.195430] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=7][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:20.195448] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=17][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:20.195454] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:20.195467] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:20.195474] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.195482] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=6][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.195488] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:20.195494] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:20.195501] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:20.195506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.195538] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=29][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:20.195550] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.195555] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.195559] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:20.195574] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10480][Occam][T1][YB42AC175881-000616AB083FEC67-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:20.195584] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10480][Occam][T0][YB42AC175881-000616AB083FEC67-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:20.195590] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)) [2024-04-22 16:23:20.195610] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:20.195621] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:20.195627] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)") [2024-04-22 16:23:20.195633] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd148e050, start=1713774200194524, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)) [2024-04-22 16:23:20.195684] WDIAG [SHARE] operator() (ob_event_history_table_operator.cpp:393) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] sync execute sql failed(sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774181765452), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'FOLLOWER ACTIVE', 'CURRENT', 'LEADER RECONFIRM', '', 0, '', 0, 'leader:"172.23.88.129:2882", new_leader_epoch:10', '172.23.88.129', 2882)", ret=-5019) [2024-04-22 16:23:20.195921] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.196284] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=7][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:20.196320] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=29] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:20.196332] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:20.196349] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:20.196358] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:20.196365] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:20.196369] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:20.196391] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=19][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:20.196395] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:20.196400] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=4][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:20.196405] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=5][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:20.196411] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=6][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:20.196416] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:20.196424] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:20.196445] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.196458] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.196462] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:20.196468] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:20.196475] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:20.196480] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.196489] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=8][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:20.196499] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.196502] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.196506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:20.196514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10480][Occam][T1][YB42AC175881-000616AB083FEC68-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:20.196525] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10480][Occam][T0][YB42AC175881-000616AB083FEC68-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:20.196531] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882)) [2024-04-22 16:23:20.196545] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:20.196551] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:20.196574] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882)") [2024-04-22 16:23:20.196588] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd1460050, start=1713774200195700, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882)) [2024-04-22 16:23:20.196613] WDIAG [SHARE] operator() (ob_event_history_table_operator.cpp:393) [10480][Occam][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] sync execute sql failed(sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774182165869), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER RECONFIRM', 'CURRENT', 'LEADER ACTIVE', '', 0, '', 0, 'reconfirm_stage_cost:400327, reconfirm_to_active_cost:400520, config_version:{proposal_id:7, config_seq:8}', '172.23.88.129', 2882)", ret=-5019) [2024-04-22 16:23:20.196855] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=79] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.196874] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.197312] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=5][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:20.197344] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:20.197353] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:20.197360] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=7][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:20.197366] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:20.197378] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:20.197383] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:20.197389] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:20.197393] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:20.197397] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=4][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:20.197403] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=6][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:20.197407] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=3][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:20.197411] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:20.197419] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:20.197424] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.197429] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.197450] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=20][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:20.197454] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774153711483), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER ACTIVE', 'CURRENT', 'FOLLOWER PENDING', '', 0, '', 0, 'pending_end_lsn:{lsn:338140}', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:20.197464] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=9][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:20.197468] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port) VALUES (usec_to_time(1713774153711483), 'LOG', 'ROLE TRANSITION', 'TENANT_ID', 1, 'LS_ID', 1, 'PREVIOUS', 'LEADER ACTIVE', 'CURRENT', 'FOLLOWER PENDING', '', 0, '', 0, 'pending_end_lsn:{lsn:338140}', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.197477] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=7][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:20.197487] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.197491] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.197494] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10480][Occam][T1][YB42AC175881-000616AB083FEC69-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:20.198160] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10480][Occam][T1][YB42AC175881-000616AB083FEC6A-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:20.198189] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10480][Occam][T1][YB42AC175881-000616AB083FEC6A-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:20.198579] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.199038] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:20.199161] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:20.199201] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:20.199228] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:20.199250] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:20.205633] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.205721] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1846074, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.206632] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:20.206733] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774199851854328, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774200206619) [2024-04-22 16:23:20.206764] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774200006566, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:20.207364] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.207486] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14619657, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=125, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.210065] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=63] PNIO [ratelimit] time: 1713774200210063, bytes: 66318166, bw: 0.029076 MB/s, add_ts: 1001919, add_bytes: 30547 [2024-04-22 16:23:20.212935] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:20.213000] WDIAG [SERVER] do_heartbeat_event (ob_heartbeat.cpp:266) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=48][errcode=0] try reload schema failed(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=-4023) [2024-04-22 16:23:20.215369] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=19][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:20.215492] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:20.215543] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=46][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:20.215568] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=22][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:20.215587] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:20.215606] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:20.215635] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=27][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:20.215779] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=134][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:20.215795] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:20.215808] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:20.215826] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=16][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:20.215856] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=27][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:20.215871] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=13][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:20.215884] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:20.215896] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:20.215937] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=28][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:20.215953] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.215968] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.215982] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:20.215993] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:20.216039] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=42][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:20.216062] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=20][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.216107] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=38][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:20.216154] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=38][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.216169] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.216179] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:20.216217] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED87-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:20.216254] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=31][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:20.216280] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=22][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:20.216297] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=16][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:20.216336] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=35][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:20.216353] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=15][errcode=-5019] query failed(ret=-5019, conn=0x7f3ceeb84050, start=1713774200215105, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:20.216378] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=22][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:20.216394] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=12][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:20.216544] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=27][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:20.216578] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED87-0-0] [lt=32][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:20.216909] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=251][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:39, tid:10410}]) [2024-04-22 16:23:20.223372] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.223490] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=15000, remain_us=1828305, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.231190] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=11] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:20.231955] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=67][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:20.231979] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=17] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:20.232000] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=19][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:20.232008] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=7][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:20.232015] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=7][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:20.232025] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:20.232052] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:20.232069] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=14][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:20.232077] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=7][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:20.232083] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=6][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:20.232088] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:20.232093] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:20.232099] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:20.232103] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:20.232109] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:20.232121] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=7][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:20.232125] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.232131] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:20.232136] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:20.232143] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:20.232148] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:20.232153] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.232163] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=8][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:20.232173] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.232177] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:20.232181] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:20.232190] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:20.232196] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:20.232202] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:20.232208] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=6][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:20.232213] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:20.232218] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45f0050, start=1713774200231850, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:20.232226] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=8][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:20.232231] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:20.232273] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=6][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:20.232280] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=6][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:20.232293] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=12][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:20.232298] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=5][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:20.232304] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=4][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:20.232311] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=6][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:20.232318] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED83-0-0] [lt=7][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:20.236556] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A9-0-0] [lt=443][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:20.236624] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A9-0-0] [lt=64][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:20.236722] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A9-0-0] [lt=92][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:20.236776] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35A9-0-0] [lt=51][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:20.239309] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=59] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:20.240584] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.240668] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=16000, remain_us=1811127, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.245903] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFF5B-0-0] [lt=69][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:20.247343] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=57] PNIO [ratelimit] time: 1713774200247341, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007608, add_bytes: 0 [2024-04-22 16:23:20.260204] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.260288] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=17000, remain_us=1791507, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.274889] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:20.280927] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.281005] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=18000, remain_us=1770790, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.295109] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:20.302618] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.302701] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=1749094, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.308923] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.308983] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309011] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309036] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309085] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309106] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309129] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309142] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309155] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309168] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309182] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309195] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309208] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309221] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309245] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.309259] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:20.311681] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.311773] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14515370, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=126, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.321883] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:20.325834] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.325914] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1725881, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.349995] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.350080] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=21000, remain_us=1701715, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.374744] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.374828] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=22000, remain_us=1676967, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.400591] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.400703] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=23000, remain_us=1651093, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.406898] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:20.406973] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774200140203387, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774200406884) [2024-04-22 16:23:20.407005] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774200206785, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:20.415421] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.415537] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14411605, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=127, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.427686] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.427831] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=24000, remain_us=1623964, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.439449] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:20.455871] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.455953] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=25000, remain_us=1595842, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.456098] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C4E-0-0] [lt=83] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=1, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:20.456188] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C4E-0-0] [lt=60] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=6, avg_log_batch_cnt=0, total_group_log_size=732, avg_group_log_size=122) [2024-04-22 16:23:20.456224] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C4E-0-0] [lt=32] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=732) [2024-04-22 16:23:20.475204] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:20.484134] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.484217] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=26000, remain_us=1567578, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.512587] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.512673] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=27000, remain_us=1539122, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.519404] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.519589] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14307553, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=128, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.543544] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.543586] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFD23-0-0] [lt=50][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774200543511}) [2024-04-22 16:23:20.543629] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=28000, remain_us=1508166, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.567787] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB091C3078-0-0] [lt=26][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2976, log_task={header:{begin_lsn:{lsn:362758}, end_lsn:{lsn:362880}, log_id:2976, min_scn:{val:1713774200452112599, v:0}, max_scn:{val:1713774200452112599, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:362636}, prev_proposal_id:8, committed_end_lsn:{lsn:362758}, data_checksum:667309872, accum_checksum:3635085703, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774200456086, freeze_ts:1713774200456086, submit_ts:1713774200456162, flushed_ts:1713774200460773, gen_to_freeze cost time:0, gen_to_submit cost time:76, submit_to_flush cost time:4611}, fs_cb_begin_ts=1713774200567770, log_life_time=111684) [2024-04-22 16:23:20.571641] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=860) [2024-04-22 16:23:20.573629] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.573766] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=29000, remain_us=1478030, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.606324] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.606382] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=30000, remain_us=1445413, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.607137] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:20.607185] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:20.607222] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774200346152509, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774200607125) [2024-04-22 16:23:20.607245] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774200407053, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:20.623784] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.623841] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14203302, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=129, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.639595] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:20.640581] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.640750] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=31000, remain_us=1411045, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.675993] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:20.676545] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.676603] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=32000, remain_us=1375192, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.707225] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:20.712399] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.712532] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=33000, remain_us=1339263, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.721127] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=52] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:20.727868] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.727967] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14099176, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=130, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.741664] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35AC-0-0] [lt=191][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:20.741698] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35AC-0-0] [lt=32][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:20.741729] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35AC-0-0] [lt=28][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:20.741757] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35AC-0-0] [lt=26][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:20.750986] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.751074] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1300721, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.756785] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C50-0-0] [lt=50][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2977, log_task={header:{begin_lsn:{lsn:362880}, end_lsn:{lsn:363008}, log_id:2977, min_scn:{val:1713774200564621708, v:0}, max_scn:{val:1713774200564621708, v:0}, data_len:72, proposal_id:8, prev_lsn:{lsn:362758}, prev_proposal_id:8, committed_end_lsn:{lsn:362880}, data_checksum:1016883040, accum_checksum:2331855686, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774200567737, freeze_ts:1713774200567737, submit_ts:1713774200567747, flushed_ts:1713774200571775, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:4028}, fs_cb_begin_ts=1713774200756754, log_life_time=189017) [2024-04-22 16:23:20.756993] INFO [STORAGE] update_id_meta (ob_ls_meta.cpp:721) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=117] update id meta(ret=0, service_type=0, limited_id=1713774217210958129, latest_scn={val:1713774200564621708, v:0}, *this={tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774217210958129, latest_log_ts:{val:1713774200564621708, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}) [2024-04-22 16:23:20.757159] INFO [STORAGE.TRANS] handle_replay_result (ob_id_service.cpp:237) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=162] handle reply result(ret=0, service_type_=0, last_id=1713774217210958129, limited_id=1713774217210958129, log_ts={val:1713774200564621708, v:0}) [2024-04-22 16:23:20.757185] INFO [STORAGE.TRANS] replay (ob_id_service.cpp:210) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=25] ObIDService replay(service_type_=0, ret=0, ls_log={last_id_:1713774217210958129, limit_id_:1713774217210958129}, base_header={version:1, log_type:7, flag:0, replay_hint:0}) [2024-04-22 16:23:20.763148] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=80] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=4883, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:20.763246] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=69] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=73, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:20.763415] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=136] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=79, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:20.789330] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.789556] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1262239, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.807341] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=152][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:20.807491] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774200452112599, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774200807325) [2024-04-22 16:23:20.807527] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774200607261, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:20.818789] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=59][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:565, tid:10476}]) [2024-04-22 16:23:20.824729] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:35, local_retry_times:35, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.824832] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=79][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.824854] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.824865] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.824892] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.828035] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:20.828086] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=49][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.828120] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.828144] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.828131] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:130, local_retry_times:130, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.828161] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.828216] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.828239] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=76][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.828245] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.828255] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.828261] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.828291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.828306] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.828346] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.828360] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.828377] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=36, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.828413] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19] will sleep(sleep_us=36000, remain_us=1223383, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.831757] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] [2024-04-22 16:23:20.831830] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=69][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.831874] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.832012] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=133][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.832040] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.832060] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.832078] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.832149] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.832217] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=59][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.832240] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:20.832261] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=131, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.832297] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=100000, remain_us=13994846, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=131, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.839814] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=76] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:20.864603] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:36, local_retry_times:36, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.864718] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=93][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.864746] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.864761] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.864863] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=76][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.868968] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=34][errcode=-4725] [2024-04-22 16:23:20.869010] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.869040] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.869063] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.869079] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.869090] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.869101] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.869149] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.869202] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.869217] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.869234] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=37, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.869293] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=42] will sleep(sleep_us=37000, remain_us=1182503, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.876691] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=89] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:20.906464] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:37, local_retry_times:37, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.906546] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=59][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.906566] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.906576] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.906600] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.910518] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:20.910556] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=36][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.910585] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.910601] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.910692] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=88][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.910718] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.910735] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.910796] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.910844] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=39][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.910861] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.910877] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=38, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.910908] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16] will sleep(sleep_us=38000, remain_us=1140887, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.932477] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:131, local_retry_times:131, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.932641] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=123][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.932673] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.932693] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.932725] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.937101] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] [2024-04-22 16:23:20.937144] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.937175] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.937194] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.937209] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.937221] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.937232] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.937287] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.937352] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=57][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.937367] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:20.937386] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=132, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.937423] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=13889720, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=132, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:20.949056] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:38, local_retry_times:38, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.949150] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.949178] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.949285] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=104][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.949324] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.951878] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=27][errcode=-4725] [2024-04-22 16:23:20.951914] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=34][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.951942] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.951964] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.951986] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.951999] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.952011] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.952056] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.952095] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.952108] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.952154] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=43][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=39, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.952193] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23] will sleep(sleep_us=39000, remain_us=1099602, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:20.991355] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:39, local_retry_times:39, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:20.991519] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=134][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.991547] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:20.991566] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:20.991679] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:20.995814] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=76][errcode=-4725] [2024-04-22 16:23:20.995854] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:20.995883] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:20.995901] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:20.995970] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=67][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:20.995983] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:20.995994] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.996080] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:20.996140] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=52][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:20.996163] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:20.996183] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=40, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:20.996222] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20] will sleep(sleep_us=40000, remain_us=1055574, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.007595] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:21.007731] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=131][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774200750171707, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774201007581) [2024-04-22 16:23:21.007764] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774200807549, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:21.036490] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=71][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:40, local_retry_times:40, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.036604] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=86][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.036627] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.036879] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=244][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.036933] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=36][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.037617] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:132, local_retry_times:132, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.037759] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=81][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.037790] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.037809] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.037837] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.040891] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=25][errcode=-4725] [2024-04-22 16:23:21.040943] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=49][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:21.040975] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:21.040994] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:21.041009] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:21.041020] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:21.041031] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.041078] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.041119] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=34][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.041133] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:21.041173] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=41, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.041209] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20] will sleep(sleep_us=41000, remain_us=1010586, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.041673] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:21.041721] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:21.041859] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=134][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:21.041884] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:21.041958] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=71][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:21.041978] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:21.041992] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.042044] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.042102] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.042124] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:21.042148] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=133, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.042196] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28] will sleep(sleep_us=100000, remain_us=13784947, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=133, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.050030] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=67] Cache replace map node details(ret=0, replace_node_count=0, replace_time=10031, replace_start_pos=1069538, replace_num=62914) [2024-04-22 16:23:21.050077] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:21.064015] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF275-0-0] [lt=95] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=25, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:21.064060] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF275-0-0] [lt=44] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=55000, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:21.064080] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF275-0-0] [lt=18] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=23, avg_log_submit_to_flush_time=5433, avg_log_submit_to_slide_time=54977) [2024-04-22 16:23:21.064086] INFO [PALF] read (log_cache.cpp:100) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=196] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=982, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:21.064267] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=19] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=106, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:21.064338] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=61] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=39, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:21.077368] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=69] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:21.082388] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:41, local_retry_times:41, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.082552] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=129][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.082585] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=30][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.082597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.082690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.089561] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=95][errcode=-4725] [2024-04-22 16:23:21.089606] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=42][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:21.089635] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:21.089653] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:21.089666] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:21.089676] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:21.089688] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.089759] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.089818] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=52][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.089839] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:21.089858] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=42, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.090062] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=180] will sleep(sleep_us=42000, remain_us=961735, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.106084] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=64] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:21.132218] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=39][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:42, local_retry_times:42, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.132301] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.132326] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.132341] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.132368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.135981] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] [2024-04-22 16:23:21.136012] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=29][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:21.136034] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:21.136051] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:21.136064] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:21.136073] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:21.136098] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.136135] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.136179] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=38][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.136195] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:21.136210] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=43, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.136238] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14] will sleep(sleep_us=43000, remain_us=915557, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.142379] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=53][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:133, local_retry_times:133, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.142505] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=96][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.142535] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.142549] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.142574] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.146388] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [2024-04-22 16:23:21.146471] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=48][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:21.146571] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=96][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:21.146605] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:21.146637] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:21.146666] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:21.146679] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.146731] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.146768] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.146845] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=74][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:21.146866] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=134, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.146899] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=13680243, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=134, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.179370] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:43, local_retry_times:43, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.179484] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=91][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.179506] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.179517] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.179563] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.183080] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=50][errcode=-4725] [2024-04-22 16:23:21.183119] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:21.183148] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:21.183165] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:21.183178] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:21.183189] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:21.183200] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.183246] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.183299] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.183320] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:21.183339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=44, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.183400] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=44] will sleep(sleep_us=44000, remain_us=868395, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.189509] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=49] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.190494] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.190581] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=79] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.191144] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.191171] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.191278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.191570] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=178] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.191852] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=68] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.191854] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.191999] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.192118] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.192620] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=56] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.192612] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=43] gc stale ls task succ [2024-04-22 16:23:21.192847] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=51] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.193499] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.193780] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:21.193858] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=34] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=10790, mem_used=79030272) [2024-04-22 16:23:21.193908] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.193936] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:21.194168] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.194316] INFO pn_ratelimit (group.c:589) [10608][IngressService][T0][Y0-0000000000000000-0-0] [lt=42] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:23:21.194426] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.194951] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=48] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.195491] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.195961] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=64] current monitor number(seq_=-1) [2024-04-22 16:23:21.196067] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.196129] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=49] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.197874] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.198592] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.198773] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=62] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:21.199081] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=23][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:21.199172] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=67] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:21.199205] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:23:21.199225] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:21.199244] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:23:21.199263] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:21.199276] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:21.199293] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:23:21.199306] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:21.199318] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:21.199329] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:21.199340] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=10][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:23:21.199351] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=10][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:23:21.199383] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=31][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:23:21.199393] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:21.199405] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:21.199415] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:21.199426] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:21.199481] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=44][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:21.199494] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:21.199507] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:21.199518] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:21.199528] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:23:21.199542] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:21.199564] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=20][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.199587] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=18][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:21.199610] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:21.199620] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:21.199628] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:21.199648] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC52-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:21.199665] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10899][BlackListServic][T0][YB42AC175881-000616AB07BFEC52-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:23:21.199679] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:21.199700] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:21.199712] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:21.199732] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1498050, start=1713774201198683, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:21.199757] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:21.199756] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:21.199773] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:21.199798] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:21.199819] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:21.199838] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:21.199875] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=19] ls blacklist refresh finish(cost_time=3162) [2024-04-22 16:23:21.207134] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=48] ====== tenant freeze timer task ====== [2024-04-22 16:23:21.207790] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:21.207835] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774201052914621, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774201207780) [2024-04-22 16:23:21.207856] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774201007786, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:21.215172] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1713774201215168, bytes: 66347544, bw: 0.027875 MB/s, add_ts: 1005105, add_bytes: 29378 [2024-04-22 16:23:21.218911] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=12][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:21.219008] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=69] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:21.219047] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=34][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:21.219164] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=113][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:21.219205] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=38][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:21.219226] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=15][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:21.219238] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:21.219254] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:21.219266] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:21.219281] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=14][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:21.219297] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=14][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:21.219312] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=13][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:21.219326] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=13][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:21.219342] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=14][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:21.219357] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=14][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:21.219465] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=60][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:21.219503] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=33][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:21.219525] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:21.219544] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=17][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:21.219562] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=14][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:21.219578] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:21.219590] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.219616] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:21.219643] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:21.219660] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:21.219679] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=17][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:21.219704] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED88-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:21.219773] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=64][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:21.219791] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:21.219808] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=15][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:21.219844] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=33][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:21.219865] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=19][errcode=-5019] query failed(ret=-5019, conn=0x7f3cbe430050, start=1713774201218662, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:21.219891] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=24][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:21.219911] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=15][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:21.220039] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=36][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:21.220066] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED88-0-0] [lt=25][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:21.222526] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=51] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713774200721045], gts:1713774200718784685, latest_srr:[mts=1713774201222466]}) [2024-04-22 16:23:21.227537] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:44, local_retry_times:44, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.227646] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=83][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.227675] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.227693] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.227727] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.231109] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.231253] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=45000, remain_us=820543, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.233621] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=4] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:21.235715] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=258][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:21.235780] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=48] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:21.235817] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=33][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:21.235846] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=26][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:21.235884] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=34][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:21.235915] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=23][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:21.235934] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=17][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:21.235958] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=15][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:21.235978] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=17][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:21.235990] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:21.236001] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:21.236012] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:21.236045] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=31][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:21.236057] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:21.236069] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:21.236093] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:21.236106] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:21.236120] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:21.236132] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:21.236144] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:21.236158] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:21.236170] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.236195] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:21.236238] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=35][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:21.236252] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:21.236262] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:21.236283] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:21.236299] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:21.236315] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:21.236328] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:21.236340] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:21.236354] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cf3bc0050, start=1713774201235517, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:21.236372] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:21.236384] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:21.236542] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=27][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:21.236595] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=49][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:21.236627] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=29][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:21.236650] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=22][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:21.236674] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=18][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:21.236697] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=19][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:21.236715] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED84-0-0] [lt=17][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:21.237162] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35AF-0-0] [lt=155][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:21.237226] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35AF-0-0] [lt=60][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:21.237274] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35AF-0-0] [lt=45][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:21.237326] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35AF-0-0] [lt=50][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:21.247045] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:21.250241] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:21.251062] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0CFFFFBE-0-0] [lt=64][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:21.251185] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.251254] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13575889, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=135, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.254970] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1713774201254966, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007625, add_bytes: 0 [2024-04-22 16:23:21.277813] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:21.280092] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.280179] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=46000, remain_us=771617, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.307913] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:21.308107] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=149][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774201052914621, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774201307896) [2024-04-22 16:23:21.308144] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774201207870, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:21.308621] INFO [PALF] check_parent_health (log_config_mgr.cpp:2764) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=41] re_register_parent reason(palf_id=1, self="172.23.88.129:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713774197304035, last_first_register_time_us=1713773712050886, register_time_us=1713774197304035, curr_time_us=1713774201308617) [2024-04-22 16:23:21.308756] INFO [PALF] check_parent_health (log_config_mgr.cpp:2769) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=67] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:21.310863] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.310943] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=77] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.310971] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.310993] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311009] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311022] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311043] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311064] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311083] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311106] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311143] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=35] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311159] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311180] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311200] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311221] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.311241] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:21.326321] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:21.330263] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.330365] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=47000, remain_us=721430, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.355583] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.355675] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13471467, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=136, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.381334] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.381423] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=48000, remain_us=670373, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.433018] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.433110] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=49000, remain_us=618686, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.450397] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=63] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:21.459071] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.459193] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=13367950, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=137, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.478314] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:21.485879] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.485969] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=50000, remain_us=565827, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.493258] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=56] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=14885, clean_start_pos=4404015, clean_num=125829) [2024-04-22 16:23:21.508100] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:21.508165] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774201353611942, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774201508087) [2024-04-22 16:23:21.508193] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774201308172, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:21.540122] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.540215] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=51000, remain_us=511580, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.557663] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C5A-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:21.557785] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C5A-0-0] [lt=60] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=860, avg_group_log_size=122) [2024-04-22 16:23:21.557837] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C5A-0-0] [lt=46] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=860) [2024-04-22 16:23:21.563428] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.563510] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFD46-0-0] [lt=42][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774201563386}) [2024-04-22 16:23:21.563591] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=13263552, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=138, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.595428] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.595554] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=52000, remain_us=456242, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.650553] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:21.651128] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.651226] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=53000, remain_us=400571, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.662992] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=854) [2024-04-22 16:23:21.667027] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.667118] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=13160025, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=139, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.693857] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:21.707656] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.707765] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=54000, remain_us=344030, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.708328] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:21.708385] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:21.708423] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774201547099215, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774201708315) [2024-04-22 16:23:21.708484] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774201508229, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:21.708516] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:21.723626] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:21.743414] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B2-0-0] [lt=229][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:21.743461] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B2-0-0] [lt=45][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:21.743512] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B2-0-0] [lt=33][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:21.743530] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B2-0-0] [lt=17][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:21.763839] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=96] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=5777, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:21.763919] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=113, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:21.764032] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=106] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=91, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:21.766149] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.766263] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=55000, remain_us=285533, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.771031] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.771117] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=13056025, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=140, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.791207] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=97] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:21.825819] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.825918] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=1] will sleep(sleep_us=56000, remain_us=225878, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.850765] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=64] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:21.875268] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.875352] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12951790, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=141, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:21.886070] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.886172] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=0] will sleep(sleep_us=57000, remain_us=165624, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.894650] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=52] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:21.908531] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:21.908635] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774201752622191, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774201908517) [2024-04-22 16:23:21.908667] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774201708504, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:21.908808] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=123][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:21.908881] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:21.908913] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:21.908933] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:21.908976] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=40][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:21.908995] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:21.909008] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:21.909025] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:23:21.909037] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:21.909048] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:21.909066] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=16][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:21.909083] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=16][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:21.909101] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=16][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:21.909119] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=15][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:21.909135] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:21.909170] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=20][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:21.909185] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:21.909213] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=24][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:21.909225] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:21.909237] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882, ret=-5019) [2024-04-22 16:23:21.909252] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:21.909264] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.909291] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:21.909318] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:21.909329] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:21.909339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:21.909362] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:21.909377] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC14-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-04-22 16:23:21.909401] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:21.909416] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:21.909429] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:21.909484] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-5019] query failed(ret=-5019, conn=0x7f3d1177a050, start=1713774201908604, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:21.909510] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:21.909529] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f3ce8856298, table=__all_server, condition=where svr_ip='172.23.88.129' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882, columns_str="zone") [2024-04-22 16:23:21.909570] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:21.909681] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:635) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f3ce8856298, where_condition="where svr_ip='172.23.88.129' and svr_port=2882", zone_name_holder=) [2024-04-22 16:23:21.909700] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:568) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:23:21.909716] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:577) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:23:21.909761] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:21.921635] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=63][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:295, tid:10476}]) [2024-04-22 16:23:21.943426] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=34][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:57, local_retry_times:57, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.943570] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=119][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.943592] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.943604] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.943630] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.947611] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=62][errcode=-4725] [2024-04-22 16:23:21.947657] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:21.947684] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:21.947700] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:21.947715] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:21.947727] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:21.947738] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.947785] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.947839] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.947862] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:21.947888] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=58, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.947962] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=52] will sleep(sleep_us=58000, remain_us=103834, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713774202051794) [2024-04-22 16:23:21.975508] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:141, local_retry_times:141, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:21.975615] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=72][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.975646] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:21.975665] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:21.975700] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:21.980389] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] [2024-04-22 16:23:21.980474] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=81][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:21.980504] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:21.980521] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:21.980538] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:21.980550] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:21.980562] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.980645] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:21.980713] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=58][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:21.980737] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:21.980760] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=142, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:21.980801] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=100000, remain_us=12846341, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=142, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.006112] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=22][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:58, local_retry_times:58, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.006207] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=69][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.006236] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.006255] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.006308] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.010390] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=21][errcode=-4012] [2024-04-22 16:23:22.010464] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=42][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.010490] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=24][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:23:22.010505] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.010520] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=14][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:23:22.010531] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:23:22.010563] WDIAG [SQL] open (ob_result_set.cpp:160) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=30][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:22.010575] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:22.010588] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.010605] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.010628] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:23:22.010659] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=20][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:22.010670] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:22.010680] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=10][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:23:22.010702] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10476][SerScheQueue0][T1][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=11][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:23:22.010722] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1957131) [2024-04-22 16:23:22.010738] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:23:22.010823] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=80][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:22.010848] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=23][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:23:22.010864] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=15][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:22.010879] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:22.010903] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:22.011005] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=17][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:23:22.011037] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=29][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:22.011059] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=19][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:23:22.011074] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=13][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:22.011094] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:22.011129] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:521) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=32][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-04-22 16:23:22.011149] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:236) [10476][SerScheQueue0][T0][YB42AC175881-000616AB0ABFEC0B-0-0] [lt=18][errcode=-4012] fail to process refresh task(ret=-4012, ret="OB_TIMEOUT", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-04-22 16:23:22.011174] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:599) [10476][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4012] fail to batch process task(ret=-4012) [2024-04-22 16:23:22.011186] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:499) [10476][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] fail to batch execute task(ret=-4012, tasks.count()=1) [2024-04-22 16:23:22.011254] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:261) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:22.011283] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:305) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24] [REFRESH_SCHEMA] begin refresh schema, (begin_time=1713774202011282) [2024-04-22 16:23:22.011292] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2388) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[]) [2024-04-22 16:23:22.014933] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] [2024-04-22 16:23:22.014954] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.014966] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.014976] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.014985] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.014991] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.014996] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.015017] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.015031] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.015034] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=3][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.015040] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.015052] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5] already timeout, do not need sleep(sleep_us=0, remain_us=1996256, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.015062] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.015072] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.015076] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=3][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.015078] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=2][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.015089] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=3][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.018293] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] [2024-04-22 16:23:22.018304] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.018312] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.018318] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.018324] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.018329] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.018334] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.018355] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.018366] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.018372] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.018379] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.018390] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6] will sleep(sleep_us=1000, remain_us=1992919, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.019463] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.019479] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.019485] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.019490] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.019498] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.021795] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] [2024-04-22 16:23:22.021811] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.021820] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=8][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.021827] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.021833] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.021838] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.021843] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.021861] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.021872] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.021876] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=3][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.021881] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.021898] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12] will sleep(sleep_us=2000, remain_us=1989410, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.023975] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.023992] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.023998] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.024002] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.024016] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.026266] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] [2024-04-22 16:23:22.026277] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.026285] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.026290] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.026296] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.026301] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.026312] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.026328] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.026339] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.026342] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=2][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.026347] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=3, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.026356] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4] will sleep(sleep_us=3000, remain_us=1984953, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.029446] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:3, local_retry_times:3, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.029469] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.029478] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=8][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.029484] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.029496] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.031663] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] [2024-04-22 16:23:22.031679] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.031689] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.031697] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.031705] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.031712] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.031716] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.031734] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.031747] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.031752] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.031758] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=4, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.031777] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13] will sleep(sleep_us=4000, remain_us=1979531, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.035863] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:4, local_retry_times:4, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.035886] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.035894] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.035901] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.035912] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.038231] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] [2024-04-22 16:23:22.038247] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.038257] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.038265] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.038273] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.038280] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.038294] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.038315] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.038328] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.038333] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=4][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.038339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=5][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=5, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.038351] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6] will sleep(sleep_us=5000, remain_us=1972958, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.043571] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=6][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:5, local_retry_times:5, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.043653] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.043674] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.043685] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.043729] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.048212] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=65][errcode=-4725] [2024-04-22 16:23:22.048265] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=50][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.048297] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.048315] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.048329] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.048340] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.048355] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.048416] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.048558] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=134][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.048575] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.048593] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=6, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.048666] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=56] will sleep(sleep_us=6000, remain_us=1962644, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.050953] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=57] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:22.054878] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=89][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:6, local_retry_times:6, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.054959] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=51][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.054992] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.055005] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.055028] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.058587] INFO [COMMON] handle (memory_dump.cpp:468) [10428][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=24] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-04-22 16:23:22.058692] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] [2024-04-22 16:23:22.058742] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=48][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.058780] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=34][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.058803] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.058845] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=40][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.058857] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.058868] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.058915] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.058973] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=49][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.059003] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.059031] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=7, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.059070] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26] will sleep(sleep_us=7000, remain_us=1952239, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.066194] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:7, local_retry_times:7, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.066252] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=42][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.066273] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.066303] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=28][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.066334] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.069810] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] [2024-04-22 16:23:22.069845] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=33][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:22.069871] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:22.069892] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:22.069907] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:22.069917] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:22.069927] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.069969] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.070011] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=35][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:22.070025] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:22.070058] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=8, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.070090] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18] will sleep(sleep_us=8000, remain_us=1941220, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.078226] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:8, local_retry_times:8, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.078320] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=70][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.078404] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=81][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:22.078424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:22.078486] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=48][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:22.081064] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:142, local_retry_times:142, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:22.082668] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:22.082754] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.082820] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=1928490, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.084729] INFO handle (memory_dump.cpp:589) [10428][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=137] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 135 206 0 1 3 1 1 0 1 4 1 1 0 1 5 1 4 0 1 7 1 2 0 1 8 62 1 0 1 12 1 2 0 1 14 1 1 0 1 16 1 1 0 1 18 2 2 0 500 0 55 211 0 500 7 3 3 0 500 8 55 1 0 500 9 3 1 0 500 10 7 2 0 500 17 12 7 0 500 22 3 48 0 500 23 16 9 0 508 0 4 8 0 508 8 15 1 0 509 0 3 8 0 509 8 9 1 0 cost_time: 25561 [2024-04-22 16:23:22.084826] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.084906] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12742236, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=143, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.094990] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=72] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:22.096768] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.096873] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=10000, remain_us=1914437, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.107114] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=35] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:22.107932] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED42-0-0] [lt=62] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:22.108173] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED44-0-0] [lt=127] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:22.108782] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:22.108914] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=127][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774201752622191, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774202108768) [2024-04-22 16:23:22.108953] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774201908688, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:22.110982] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.111079] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=11000, remain_us=1900231, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.124779] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.124880] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=12000, remain_us=1886429, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.140025] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.140111] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=1871199, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.157134] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.157218] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1854091, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.158644] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C62-0-0] [lt=26] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=29, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:22.158702] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C62-0-0] [lt=56] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=40131, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:22.158723] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C62-0-0] [lt=20] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=23, avg_log_submit_to_flush_time=5671, avg_log_submit_to_slide_time=40107) [2024-04-22 16:23:22.158713] INFO [PALF] read (log_cache.cpp:100) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=732, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:22.158748] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C62-0-0] [lt=23][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2986, log_task={header:{begin_lsn:{lsn:363984}, end_lsn:{lsn:364106}, log_id:2986, min_scn:{val:1713774201951475069, v:0}, max_scn:{val:1713774201951475069, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:363862}, prev_proposal_id:8, committed_end_lsn:{lsn:363984}, data_checksum:667309872, accum_checksum:433645594, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774201957903, freeze_ts:1713774201957903, submit_ts:1713774201957916, flushed_ts:1713774201962352, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:4436}, fs_cb_begin_ts=1713774202158611, log_life_time=200708) [2024-04-22 16:23:22.158887] INFO [LIB] stat (utility.h:1165) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=38] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=98, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:22.158966] INFO [LIB] stat (utility.h:1165) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=76] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=10, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:22.177101] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.177189] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=1834120, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.189879] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.189985] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12637157, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=144, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.189973] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.190163] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.190232] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.190291] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.190640] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=53] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.190791] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [10838][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=25] XA scheduler heartbeat task statistics(avg_time=1) [2024-04-22 16:23:22.190800] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=49] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.191500] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=78] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.191710] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.191899] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.191924] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.191929] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC12-0-0] [lt=18][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-04-22 16:23:22.191958] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC12-0-0] [lt=27][errcode=-4002] refresh sever list failed(ret=-4002) [2024-04-22 16:23:22.191973] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC12-0-0] [lt=14][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-04-22 16:23:22.191969] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.192211] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.192722] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=37] gc stale ls task succ [2024-04-22 16:23:22.192717] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:406) [10819][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=668] finish to resize log external storage handler(current_ts=1713774202192705, tenant_max_cpu=3, valid_ls_count=0, other_ls_count=0, new_concurrency=0) [2024-04-22 16:23:22.193281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=68] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.193614] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=36] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.193999] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=50] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:22.194025] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=18] ====== check clog disk timer task ====== [2024-04-22 16:23:22.194035] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.194074] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=36] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:22.194103] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=70] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=10836, mem_used=79030272) [2024-04-22 16:23:22.194131] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=28] cannot_recycle_log_size statistics(cannot_recycle_log_size=364106, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:22.194137] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=58] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:22.194302] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=21] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-22 16:23:22.194336] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=48] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.194377] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=59] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=100) [2024-04-22 16:23:22.194365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=67] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.194532] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=13] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-22 16:23:22.194574] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=33] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-22 16:23:22.194605] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=25] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=100) [2024-04-22 16:23:22.194800] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:306) [10844][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=16] ====== traversal_flush timer task ====== [2024-04-22 16:23:22.195041] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.194945] INFO [MDS] flush (mds_table_mgr.cpp:172) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC10-0-0] [lt=38] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774217210958129, latest_log_ts:{val:1713774200564621708, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:22.195107] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:338) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC10-0-0] [lt=154] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-22 16:23:22.195901] INFO [CLOG] run1 (ob_garbage_collector.cpp:1348) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] Garbage Collector is running(seq_=52, gc_interval=10000000) [2024-04-22 16:23:22.195885] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0D-0-0] [lt=85] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:22.196263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=56] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.196257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.196421] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=55] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.197078] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:273) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=42] GC check ls in member list, skip it(id={id:1}, leader="172.23.88.130:2882") [2024-04-22 16:23:22.197125] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1441) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=37] gc_check_member_list_ cost time(ret=0, time_us=1180) [2024-04-22 16:23:22.197153] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1713) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=18] execute_gc cost time(ret=0, time_us=1) [2024-04-22 16:23:22.197181] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1636) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=16] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-22 16:23:22.197211] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1658) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=21] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-04-22 16:23:22.197233] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1713) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=20] execute_gc cost time(ret=0, time_us=23) [2024-04-22 16:23:22.197248] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] ObSafeDestroyHandler start process [2024-04-22 16:23:22.197263] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2024-04-22 16:23:22.197295] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [10823][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=16] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2024-04-22 16:23:22.197566] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.197650] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=16000, remain_us=1813659, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.197760] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=60] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.198031] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=69] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.198423] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:131) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] ====== checkpoint timer task ====== [2024-04-22 16:23:22.198517] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-22 16:23:22.199815] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:22.200304] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:22.200375] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=42][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:22.200413] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:22.200485] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:22.200494] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=49][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:22.200540] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=29] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-04-22 16:23:22.200569] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=25][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-04-22 16:23:22.200589] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-04-22 16:23:22.200608] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-04-22 16:23:22.200625] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:22.200638] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:22.200654] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-04-22 16:23:22.200667] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:22.200678] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:22.200689] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:22.200733] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=41][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:22.200748] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=13][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:22.200760] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:22.200772] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:22.200796] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:22.200810] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:22.200824] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:22.200836] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:22.200848] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:22.200862] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:22.200874] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.200900] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:22.200940] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=35][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:22.200952] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:22.200961] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:22.200993] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:22.201009] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC11-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2024-04-22 16:23:22.201024] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:23:22.201039] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:22.201051] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:22.201064] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd2268050, start=1713774202200239, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:23:22.201082] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:22.201095] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:479) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:22.201196] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:141) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2024-04-22 16:23:22.201211] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:252) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] failed to fetch max rule version(ret=-5019) [2024-04-22 16:23:22.201223] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:95) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2024-04-22 16:23:22.202199] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1426) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=74] succ to get rec scn(*this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:22.202279] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=77] tx ctx memtable get rec scn(this={ObITable:{this:0x7f3cd10be080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f3cd10be080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:22.202336] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:856) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=47] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-22 16:23:22.202478] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=33] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774217210958129, latest_log_ts:{val:1713774200564621708, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:22.202656] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:599) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=169] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1}) [2024-04-22 16:23:22.202691] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:470) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2024-04-22 16:23:22.202715] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] get rec log scn(service_type_=0, rec_log_ts={val:1713774200564621708, v:0}) [2024-04-22 16:23:22.202729] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-22 16:23:22.202743] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=2, rec_log_ts={val:1713773717210958135, v:0}) [2024-04-22 16:23:22.202766] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:164) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713773717210958135, v:0}, checkpoint_scn_in_ls_meta={val:1713773717210958135, v:0}, ls_id={id:1}, service_type="DAS_ID") [2024-04-22 16:23:22.202810] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1579) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-22 16:23:22.202851] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:185) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=38] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-22 16:23:22.202872] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:192) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=16] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-22 16:23:22.202868] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=25][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:22.202913] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=29] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:22.202939] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:22.202958] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:22.202977] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:22.202994] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:22.203006] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:22.203022] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:23:22.203056] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=32][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:22.203067] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:22.203078] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:22.203089] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:22.203100] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:22.203111] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:22.203122] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:22.203143] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:22.203155] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:22.203175] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:22.203192] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=15][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:22.203208] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=14][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-04-22 16:23:22.203293] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=80][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:22.203309] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=14][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.203334] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:22.203359] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:22.203370] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:22.203379] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:22.203401] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC45-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:22.203420] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10404][ServerGTimer][T0][YB42AC175881-000616AB07DFEC45-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-04-22 16:23:22.203463] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-04-22 16:23:22.203482] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:22.203494] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:22.203520] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd148e050, start=1713774202202683, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-04-22 16:23:22.203538] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:22.203657] WDIAG [SHARE] refresh (ob_all_server_tracer.cpp:568) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] fail to get servers_info(ret=-5019, ret="OB_TABLE_NOT_EXIST", GCTX.sql_proxy_=0x5596469c0388) [2024-04-22 16:23:22.203672] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:626) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] fail to refresh all server map(ret=-5019) [2024-04-22 16:23:22.203730] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1918) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=12] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-22 16:23:22.203763] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1459) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=27] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:22.203785] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1476) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=19] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:22.203803] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2682) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=16] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:22.208921] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713774201752622191, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=456292, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1713774202208914, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-22 16:23:22.210432] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2401) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=13] Dumping All Cache Objs(alloc_obj_list.count()=6, alloc_obj_list=[{obj_id:99, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175160}, {obj_id:100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174200}, {obj_id:101, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173640}, {obj_id:102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173992}, {obj_id:103, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:176531}, {obj_id:104, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:175513}]) [2024-04-22 16:23:22.210584] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2690) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=142] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:22.215714] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:22.215800] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=56] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:22.215829] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=28] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:23:22.218042] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=62] PNIO [ratelimit] time: 1713774202218039, bytes: 66381935, bw: 0.032704 MB/s, add_ts: 1002871, add_bytes: 34391 [2024-04-22 16:23:22.218207] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.218316] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=17000, remain_us=1792994, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.222387] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=11][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:22.222493] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=86] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:22.222540] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=42][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:22.222561] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:22.222580] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:22.222639] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=51][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:22.222655] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:22.222672] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:22.222685] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:22.222695] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:22.222706] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:22.222717] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:22.222729] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:22.222768] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=24][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:22.222819] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=39][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:22.222860] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=30][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:22.222874] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:22.222953] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=74][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:22.222966] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:22.222978] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:22.222993] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:22.223005] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.223030] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:22.223055] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:22.223066] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:22.223075] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:22.223109] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED89-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:22.223146] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=33][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:22.223179] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=30][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:22.223196] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=16][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:22.223208] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:22.223237] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=27][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1460050, start=1713774202222132, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:22.223273] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=34][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:22.223287] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=10][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:22.223395] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=28][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:22.223408] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED89-0-0] [lt=12][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:22.225173] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=85] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=150, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-22 16:23:22.238212] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=16] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:22.239915] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B5-0-0] [lt=101][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:22.240028] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B5-0-0] [lt=109][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:22.240182] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B5-0-0] [lt=147][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:22.240239] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B5-0-0] [lt=55][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:22.241390] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.241422] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=273][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:22.241523] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=18000, remain_us=1769786, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.241535] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=93] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:22.241568] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:22.241586] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:22.241603] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:22.241638] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=29][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:22.241650] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:22.241665] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:22.241677] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:22.241701] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=22][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:22.241711] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=9][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:22.241721] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:22.241769] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=45][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:22.241803] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=23][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:22.241816] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=12][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:22.241838] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:22.241857] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=15][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:22.241873] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:22.241909] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=22][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:22.241922] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:22.241936] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:22.241946] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:22.241970] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:22.241993] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:22.242003] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:22.242011] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:22.242075] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=35][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:22.242090] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:22.242104] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:22.242135] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=19][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:22.242163] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=26][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:22.242176] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd227a050, start=1713774202241193, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:22.242204] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=26][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:22.242218] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:22.242340] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=13][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:22.242376] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=33][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:22.242398] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=20][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:22.242423] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=23][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:22.242493] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=65][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:22.242534] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=37][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:22.242558] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED85-0-0] [lt=22][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:22.251122] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:22.256002] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0D1FEBCD-0-0] [lt=70][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:22.262580] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=53] PNIO [ratelimit] time: 1713774202262576, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007610, add_bytes: 0 [2024-04-22 16:23:22.263544] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.263587] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=19000, remain_us=1747722, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.286861] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.286948] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1724362, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.290124] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:22.294586] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.294669] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12532473, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=145, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.295654] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:22.306157] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=62] tx gc loop thread is running(MTL_ID()=1) [2024-04-22 16:23:22.306202] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=42] try gc retain ctx [2024-04-22 16:23:22.306244] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:748) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] get_max_applied_scn(scn={val:1713774187421454783, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:7, palf_committed_end_lsn_:{lsn:358576}, palf_committed_end_scn_:{val:1713774187421454783, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713774187421454783, v:0}}) [2024-04-22 16:23:22.306298] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1023) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44] get_min_unreplayed_log_info(lsn={lsn:364106}, scn={val:1713774201951475070, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774202158617, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:364106}, next_to_submit_scn_:{val:1713774201951475070, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:363984}, end_lsn:{lsn:364106}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774201951475069, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774201951475069, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:433645594, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774201951475069, v:0}}}}}) [2024-04-22 16:23:22.308214] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:253) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=74] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774217210958129, latest_log_ts:{val:1713774200564621708, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:22.309042] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=116][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:22.309110] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774201951475069, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774202309029) [2024-04-22 16:23:22.309140] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774202108977, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:22.309216] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1713774201951475069, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1713774202309162}, version_delta=357740) [2024-04-22 16:23:22.309718] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35] LogLoopThread round_cost_time(us)(round_cost_time=92) [2024-04-22 16:23:22.310572] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.310652] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=21000, remain_us=1700657, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.312815] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.312887] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=69] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.312912] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.312929] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.312942] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.312955] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.312968] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.312981] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.312994] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.313007] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.313020] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.313042] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.313056] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.313069] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.313082] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.313095] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:22.317170] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=70] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:22.335603] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.335687] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=22000, remain_us=1675623, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.358793] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C64-0-0] [lt=61][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2987, log_task={header:{begin_lsn:{lsn:364106}, end_lsn:{lsn:364228}, log_id:2987, min_scn:{val:1713774202153375058, v:0}, max_scn:{val:1713774202153375058, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:363984}, prev_proposal_id:8, committed_end_lsn:{lsn:364106}, data_checksum:667309872, accum_checksum:3525091408, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774202158567, freeze_ts:1713774202158567, submit_ts:1713774202158579, flushed_ts:1713774202163180, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:4601}, fs_cb_begin_ts=1713774202358766, log_life_time=200199) [2024-04-22 16:23:22.362818] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.362903] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=23000, remain_us=1648407, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.364910] INFO [PALF] inner_write_once_ (log_block_handler.cpp:411) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=21] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=368324, this={dio_aligned_buf:{buf_write_offset:3902, buf_padding_size:194, align_size:4096, aligned_buf_size:2125824, aligned_used_ts:5914, truncate_used_ts:2169}, log_block_size:67108864, dir_fd:235, io_fd:215}, aligned_buf_len=4096, aligned_buf="", aligned_block_offset=364544, buf_len=122, total_write_size_=5774, total_write_size_after_dio_=155648, ob_pwrite_used_ts=200602, count=37, each_pwrite_cost=5421) [2024-04-22 16:23:22.389693] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.389808] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1621502, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.396217] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=53][errcode=-4018] try free send task failed(ret=-4018) [2024-04-22 16:23:22.398673] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.398758] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12428385, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=146, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.417666] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.417784] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=1593525, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.446885] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.446971] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=26000, remain_us=1564338, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.451312] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=57] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:22.476705] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.476798] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=27000, remain_us=1534511, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.496301] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=64] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:22.496387] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=51] ObArchiveSender is running(thread_index=0) [2024-04-22 16:23:22.502740] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.502821] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12324321, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=147, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.507675] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.507754] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=28000, remain_us=1503555, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.509286] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:22.509391] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774202346761521, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774202509272) [2024-04-22 16:23:22.509423] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774202309161, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:22.510036] INFO [PALF] reset_broadcast_leader_ (log_state_mgr.cpp:1255) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=77] reset_broadcast_leader_(self="172.23.88.129:2882", palf_id=1, last_recv_leader_broadcast_time_us=1713774192448041, broadcast_leader="172.23.88.130:2882") [2024-04-22 16:23:22.510086] INFO [PALF] switch_state (log_state_mgr.cpp:337) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=47] switch_state(ret=0, palf_id=1, old_role=2, role=2, old_state=2, state=2, old_leader="172.23.88.130:2882", leader="172.23.88.130:2882", old_leader_epoch=11, leader_epoch=11, need_next_loop=false, diff=54) [2024-04-22 16:23:22.539815] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.539899] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=29000, remain_us=1471410, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.559038] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C66-0-0] [lt=91] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:22.559123] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C66-0-0] [lt=58] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=6, avg_log_batch_cnt=0, total_group_log_size=732, avg_group_log_size=122) [2024-04-22 16:23:22.559155] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C66-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=732) [2024-04-22 16:23:22.572781] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.572845] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFD77-0-0] [lt=61][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774202572739}) [2024-04-22 16:23:22.572914] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=30000, remain_us=1438396, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.607093] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.607190] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12219952, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=148, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.607608] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.607717] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=31000, remain_us=1403592, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.642658] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.642772] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=32000, remain_us=1368538, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.651520] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=82] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:22.678619] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.678696] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=33000, remain_us=1332613, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.696135] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [10825][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=65] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-22 16:23:22.696758] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=95] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:22.709544] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:22.709602] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:22.709704] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=97][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774202537561591, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774202709531) [2024-04-22 16:23:22.709790] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=83][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774202509487, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:22.709836] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:22.711205] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.711295] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12115847, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=149, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.715901] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.716008] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1295302, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.726878] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=53] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:22.727681] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [10464][pnio1][T0][YB42AC175881-000616AB0B5FEDDB-0-0] [lt=54] ts response task statistics(alloc_count=3678, free_count=1838) [2024-04-22 16:23:22.735714] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B8-0-0] [lt=493][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:22.735777] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B8-0-0] [lt=59][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:22.735862] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B8-0-0] [lt=80][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:22.735912] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35B8-0-0] [lt=47][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:22.753899] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.753992] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=35000, remain_us=1257318, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.759228] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C6A-0-0] [lt=17][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2990, log_task={header:{begin_lsn:{lsn:364472}, end_lsn:{lsn:364594}, log_id:2990, min_scn:{val:1713774202644935487, v:0}, max_scn:{val:1713774202644935487, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:364350}, prev_proposal_id:8, committed_end_lsn:{lsn:364472}, data_checksum:667309872, accum_checksum:2342710680, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774202659072, freeze_ts:1713774202659072, submit_ts:1713774202659085, flushed_ts:1713774202663294, gen_to_freeze cost time:0, gen_to_submit cost time:13, submit_to_flush cost time:4209}, fs_cb_begin_ts=1713774202759200, log_life_time=100128) [2024-04-22 16:23:22.763663] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=854) [2024-04-22 16:23:22.792856] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.792965] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=36000, remain_us=1218344, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.815981] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.816117] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12011026, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=150, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.832814] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.832934] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=37000, remain_us=1178376, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.851678] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:22.874016] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.874123] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=38000, remain_us=1137187, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.897656] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:22.909762] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:22.909833] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774202750192860, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774202909719) [2024-04-22 16:23:22.909886] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774202709823, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:22.915773] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.915885] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=39000, remain_us=1095425, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.921598] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.921689] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=11905454, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=151, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:22.958670] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:22.958800] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=1052510, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:22.963589] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=59] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=4596, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:22.963704] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=86] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=74, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:22.963828] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=90, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:23.002656] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.002748] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=41000, remain_us=1008561, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.024848] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:629, tid:10476}]) [2024-04-22 16:23:23.025034] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1][errcode=-4725] [2024-04-22 16:23:23.025083] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.025124] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.025151] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.025170] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.025189] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.025206] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.025298] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=43] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.025353] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.025370] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:23.025404] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=152, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.025491] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=71] will sleep(sleep_us=100000, remain_us=11801652, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=152, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.043924] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:41, local_retry_times:41, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.044022] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=77][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.044049] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.044063] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.044089] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.047722] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=71][errcode=-4725] [2024-04-22 16:23:23.047763] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.047793] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.047811] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.047825] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.047837] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.047847] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.047892] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.047952] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=51][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.047974] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.047999] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=22][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=42, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.048070] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=54] will sleep(sleep_us=42000, remain_us=963240, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.051849] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:23.090208] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=22][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:42, local_retry_times:42, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.090335] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=106][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.090355] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.090365] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.090503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.093500] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=71][errcode=-4725] [2024-04-22 16:23:23.093545] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=42][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.093576] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.093594] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.093608] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.093619] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.093652] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.093704] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.093795] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=82][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.093818] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.093838] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=43, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.093873] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19] will sleep(sleep_us=43000, remain_us=917436, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.098382] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=71] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:23.108120] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=76] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:23.110035] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:23.110142] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774202750192860, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774203110021) [2024-04-22 16:23:23.110185] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774202909908, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:23.125649] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=44][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:152, local_retry_times:152, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.125771] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.125800] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.125814] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.125859] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.129252] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] [2024-04-22 16:23:23.129294] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.129323] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.129339] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.129353] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.129364] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.129375] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.129420] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.129527] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=99][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.129549] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:23.129569] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=153, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.129623] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38] will sleep(sleep_us=100000, remain_us=11697519, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=153, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.137117] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:43, local_retry_times:43, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.137201] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.137229] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.137258] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.137283] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.140946] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=37][errcode=-4725] [2024-04-22 16:23:23.140989] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.141015] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.141064] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=46][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.141079] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.141090] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.141100] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.141145] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.141197] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=44][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.141219] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.141248] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=44, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.141346] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=82] will sleep(sleep_us=44000, remain_us=869963, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.159583] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C6E-0-0] [lt=90] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=25, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:23.159618] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C6E-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=86645, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:23.159628] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C6E-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=23, avg_log_submit_to_flush_time=4837, avg_log_submit_to_slide_time=86622) [2024-04-22 16:23:23.159640] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C6E-0-0] [lt=11][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2992, log_task={header:{begin_lsn:{lsn:364716}, end_lsn:{lsn:364838}, log_id:2992, min_scn:{val:1713774202955038307, v:0}, max_scn:{val:1713774202955038307, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:364594}, prev_proposal_id:8, committed_end_lsn:{lsn:364716}, data_checksum:667309872, accum_checksum:919999894, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774202959324, freeze_ts:1713774202959325, submit_ts:1713774202959337, flushed_ts:1713774202963740, gen_to_freeze cost time:1, gen_to_submit cost time:13, submit_to_flush cost time:4403}, fs_cb_begin_ts=1713774203159567, log_life_time=200243) [2024-04-22 16:23:23.159647] INFO [PALF] read (log_cache.cpp:100) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=732, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:23.159805] INFO [LIB] stat (utility.h:1165) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=28] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=92, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:23.159864] INFO [LIB] stat (utility.h:1165) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=57] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=10, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:23.185516] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:44, local_retry_times:44, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.185608] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.185630] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.185642] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.185689] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.189539] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=72] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.190390] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.190427] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.190696] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=56] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.190911] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.191007] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] [2024-04-22 16:23:23.191069] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=60][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.191101] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.191118] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.191133] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.191145] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.191155] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.191205] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.191215] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.191255] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.191269] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.191296] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=45, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.191360] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=40] will sleep(sleep_us=45000, remain_us=819950, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.191556] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=62] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.191625] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.191970] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.191970] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.192825] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=46] gc stale ls task succ [2024-04-22 16:23:23.192869] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.192884] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.192975] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.193251] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [10543][Blacklist][T0][Y0-0000000000000000-0-0] [lt=53] blacklist_loop exec finished(cost_time=59, is_enabled=true, send_cnt=0) [2024-04-22 16:23:23.193311] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.193758] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.194149] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.194253] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=35] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10878, mem_used=79030272) [2024-04-22 16:23:23.194252] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=64] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:23.194282] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=28] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:23.194501] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.195111] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.195421] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=49] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.195479] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=71] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.196243] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2024-04-22 16:23:23.196948] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] [COMPACTION DAG STATUS] (start_time=1713774193193927, end_time=1713774203196946, dag_status={}) [2024-04-22 16:23:23.196981] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=30] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197000] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=18] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197018] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197034] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197022] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=98] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.197045] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197057] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197079] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197091] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197102] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:23:23.197114] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197130] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197145] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197158] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197167] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197175] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197189] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197199] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197207] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197216] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197224] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197233] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197241] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197250] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197266] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197275] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197284] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197292] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197301] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197309] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197318] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197326] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197339] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197357] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197365] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197374] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197382] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197390] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197399] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197407] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197420] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197429] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197459] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=30] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197468] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197477] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197433] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.197485] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197494] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197502] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197515] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197524] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197533] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197541] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197550] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197558] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197567] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197576] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197587] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197596] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197605] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197613] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197622] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197630] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197639] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197647] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197660] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197668] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197677] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:23:23.197687] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3583) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2024-04-22 16:23:23.197694] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2024-04-22 16:23:23.197702] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2024-04-22 16:23:23.197707] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2024-04-22 16:23:23.197713] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2024-04-22 16:23:23.197718] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2024-04-22 16:23:23.197723] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2024-04-22 16:23:23.197728] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2024-04-22 16:23:23.197757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=29] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2024-04-22 16:23:23.197763] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2024-04-22 16:23:23.197770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4271) [10795][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=0) [2024-04-22 16:23:23.197930] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=61] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.199977] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:23.200923] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:23.201016] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=79][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:23.201042] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:23.201063] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:23.209186] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=41] ====== tenant freeze timer task ====== [2024-04-22 16:23:23.222258] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=55] PNIO [ratelimit] time: 1713774203222254, bytes: 66410193, bw: 0.026836 MB/s, add_ts: 1004215, add_bytes: 28258 [2024-04-22 16:23:23.225609] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=24][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:23.225726] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=90] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:23.225804] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=72][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:23.225835] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=27][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:23.225864] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=27][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:23.225891] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=19][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:23.225912] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=19][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:23.225937] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=15][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:23.225956] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=16][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:23.225972] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=15][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:23.226010] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=28][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:23.226028] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=16][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:23.226064] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=33][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:23.226081] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=15][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:23.226098] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:23.226133] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=19][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:23.226152] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=15][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:23.226173] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=16][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:23.226190] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=15][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:23.226207] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=14][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:23.226229] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=18][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:23.226247] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=16][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.226285] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=31][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:23.226325] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=32][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:23.226360] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=33][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:23.226378] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=17][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:23.226425] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8A-0-0] [lt=17][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:23.226555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=123][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:23.226591] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=31][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:23.226617] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=24][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:23.226638] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=19][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:23.226660] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=20][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45cc050, start=1713774203225254, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:23.226689] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=27][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:23.226711] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=18][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:23.226862] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=37][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:23.226903] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8A-0-0] [lt=37][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:23.229764] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:153, local_retry_times:153, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.229815] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.229829] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.229835] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.229848] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=6][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.234826] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9][errcode=-4725] [2024-04-22 16:23:23.234863] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.234886] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.234900] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.234911] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.234920] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.234928] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.234996] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.235034] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.235050] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:23.235069] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=154, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.235103] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=100000, remain_us=11592039, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=154, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.236529] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:45, local_retry_times:45, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.236610] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.236638] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.236657] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.236689] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.240083] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35BB-0-0] [lt=196][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:23.240145] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35BB-0-0] [lt=58][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:23.240194] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35BB-0-0] [lt=44][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:23.240248] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35BB-0-0] [lt=53][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:23.240264] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] [2024-04-22 16:23:23.240316] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=50][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.240343] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.240364] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.240381] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.240392] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.240403] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.240485] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.240537] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.240557] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.240581] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=22][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=46, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.240625] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26] will sleep(sleep_us=46000, remain_us=770685, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.243760] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=9] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:23.245799] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=182][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:23.245866] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=49] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:23.245897] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:23.245917] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:23.245936] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:23.245954] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:23.245967] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:23.246005] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=29][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:23.246018] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:23.246029] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:23.246041] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:23.246051] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:23.246063] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:23.246075] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:23.246087] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:23.246110] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:23.246124] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:23.246138] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:23.246150] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:23.246170] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=17][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:23.246185] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:23.246197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.246222] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:23.246246] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:23.246257] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:23.246267] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:23.246288] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:23.246303] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:23.246318] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:23.246331] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:23.246357] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=25][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:23.246371] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3ceeb84050, start=1713774203245561, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:23.246388] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:23.246400] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:23.246528] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=15][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:23.246566] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=35][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:23.246590] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=22][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:23.246605] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=14][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:23.246620] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:23.246635] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=12][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:23.246646] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED86-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:23.260765] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10483][LSSysTblUp0][T0][YB42AC175881-000616AB0D2FEBF6-0-0] [lt=84][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:23.263874] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=48] Cache replace map node details(ret=0, replace_node_count=0, replace_time=11851, replace_start_pos=1132452, replace_num=62914) [2024-04-22 16:23:23.263933] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:23.270242] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=56] PNIO [ratelimit] time: 1713774203270238, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007662, add_bytes: 0 [2024-04-22 16:23:23.286794] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=45][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:46, local_retry_times:46, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.286883] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.286906] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.286918] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.286967] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.287093] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2192) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=37] need check schema(new_schema_version=1, cached_tenant_schema_version=1, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name) [2024-04-22 16:23:23.290881] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=44][errcode=-4725] [2024-04-22 16:23:23.290924] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.290948] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=22][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.290963] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.291014] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=48][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.291034] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.291052] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.291101] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.291151] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.291172] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.291192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=47, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.291228] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19] will sleep(sleep_us=47000, remain_us=720081, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.299218] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=127] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:23.310182] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:23.310214] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774202955038307, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774203310174) [2024-04-22 16:23:23.310230] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774203110216, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:23.314800] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.314875] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=71] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.314935] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.314951] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.314965] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.314978] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.314991] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315004] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315017] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315030] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315084] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=51] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315106] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315120] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315133] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315146] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.315159] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:23.335266] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:154, local_retry_times:154, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.335367] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.335396] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.335415] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.335480] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=51][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.338369] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:47, local_retry_times:47, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.338504] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=113][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.338552] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=45][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.338587] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=33][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.338613] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.339216] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=44][errcode=-4725] [2024-04-22 16:23:23.339274] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.339309] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.339330] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.339360] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.339393] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.339411] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.339516] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.339573] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.339588] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:23.339606] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=155, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.339641] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=100000, remain_us=11487501, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=155, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.341860] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:23.341900] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.341927] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.341949] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.341970] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.341987] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.342003] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.342062] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.342128] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=56][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.342150] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.342217] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=63][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=48, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.342255] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=22] will sleep(sleep_us=48000, remain_us=669055, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.360204] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C70-0-0] [lt=25][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=2993, log_task={header:{begin_lsn:{lsn:364838}, end_lsn:{lsn:364960}, log_id:2993, min_scn:{val:1713774203154519088, v:0}, max_scn:{val:1713774203154519088, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:364716}, prev_proposal_id:8, committed_end_lsn:{lsn:364838}, data_checksum:667309872, accum_checksum:1475013493, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774203159528, freeze_ts:1713774203159528, submit_ts:1713774203159535, flushed_ts:1713774203163948, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:4413}, fs_cb_begin_ts=1713774203360178, log_life_time=200650) [2024-04-22 16:23:23.390419] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:48, local_retry_times:48, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.390544] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=100][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.390573] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.390587] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.390613] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.394359] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:23.394403] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.394471] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=64][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.394513] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=39][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.394528] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.394540] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.394551] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.394598] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.394648] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.394670] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.394690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=49, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.394724] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18] will sleep(sleep_us=49000, remain_us=616585, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.410303] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:23.410405] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=97][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774202955038307, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774203410290) [2024-04-22 16:23:23.410470] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774203310241, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:23.439799] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:155, local_retry_times:155, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.439935] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=102][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.439958] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.439969] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.439996] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.443862] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:23.443974] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=92][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.444017] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=41][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:23.444029] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:23.444064] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:23.444217] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:23.444275] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.444308] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.444329] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.444352] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.444364] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:23.444459] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=66][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.444526] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.444584] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:23.444607] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:23.444627] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=156, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.444662] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=100000, remain_us=11382481, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=156, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.448025] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] [2024-04-22 16:23:23.448066] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.448096] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:23.448113] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.448127] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:23.448138] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:23.448193] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.448259] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=50000, remain_us=563050, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.464085] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:23.499826] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=71] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:23.502489] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.502607] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=51000, remain_us=508702, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.511425] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=11542, clean_start_pos=4529844, clean_num=125829) [2024-04-22 16:23:23.544815] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:23.549135] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.549230] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11277913, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=157, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.556965] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.557060] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=52000, remain_us=454249, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.560325] INFO [LIB] stat (utility.h:1165) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C72-0-0] [lt=94] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=1, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:23.560422] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C72-0-0] [lt=68] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=6, avg_log_batch_cnt=0, total_group_log_size=732, avg_group_log_size=122) [2024-04-22 16:23:23.560508] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11430][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C72-0-0] [lt=81] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=732) [2024-04-22 16:23:23.610545] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:23.610680] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774203352715410, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774203610527) [2024-04-22 16:23:23.610724] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774203410495, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:23.610790] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFD99-0-0] [lt=55][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713774203610634}) [2024-04-22 16:23:23.616039] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.616153] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=53000, remain_us=395156, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.653007] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.653074] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11174068, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=158, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.664275] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=57] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:23.673402] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.673540] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=54000, remain_us=337769, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.710600] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:23.712053] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:23.729408] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=133] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:23.731534] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.731648] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=55000, remain_us=279661, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.735373] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35BE-0-0] [lt=262][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:23.735497] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35BE-0-0] [lt=118][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:23.735592] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35BE-0-0] [lt=89][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:23.735695] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35BE-0-0] [lt=99][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:23.757721] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.757871] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=11069272, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=159, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.789665] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.789779] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=56000, remain_us=221530, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.804694] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=87] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:23.810736] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:23.810791] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:23.810872] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=77][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774203654427240, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774203810725) [2024-04-22 16:23:23.810918] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774203610754, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:23.850153] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.850272] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=57000, remain_us=161038, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.861975] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.862075] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=10965068, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=160, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.864479] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:23.865513] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=23] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=732) [2024-04-22 16:23:23.911701] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.911794] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=58000, remain_us=99516, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713774204011308) [2024-04-22 16:23:23.912715] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:23.965752] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=4612, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:23.965940] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=166] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=99, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:23.966059] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=59] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=80, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:23.966130] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.966204] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10860939, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=161, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:23.974219] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1][errcode=-4012] [2024-04-22 16:23:23.974273] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=52][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:23.974327] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=51][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:23:23.974344] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:23.974359] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:23:23.974370] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:23:23.974382] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:23.974393] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:23.974406] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:23.974424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:23.974487] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=60][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:23:23.974522] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:23.974535] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:23.974545] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:23:23.974581] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:23:23.974601] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1962794) [2024-04-22 16:23:23.974618] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:23:23.974634] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:23.974649] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:23:23.974661] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:23.974675] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:23.974698] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:23.974797] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:23:23.974811] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:23.974833] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=21][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:23:23.974846] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:23.974865] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:23.974881] WDIAG [SHARE.SCHEMA] operator() (ob_multi_version_schema_service.cpp:2425) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4012] fail to get baseline_schema_version(ret=-4012) [2024-04-22 16:23:23.974894] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2495) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11] [REFRESH_SCHEMA] end refresh and add schema(ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], cost=1963593) [2024-04-22 16:23:23.974911] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:308) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4012] fail to refresh and add schema(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:23.974924] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:314) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12] [REFRESH_SCHEMA] end refresh schema with new mode, (ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], used time=1963641) [2024-04-22 16:23:23.980083] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.980190] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] already timeout, do not need sleep(sleep_us=0, remain_us=1994734, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774205974922) [2024-04-22 16:23:23.983769] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.983866] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=1000, remain_us=1991058, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:23.988208] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.988282] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=2000, remain_us=1986641, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:23.993945] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:23.994027] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=3000, remain_us=1980896, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.000194] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.000272] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=4000, remain_us=1974651, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.008005] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.008095] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=5000, remain_us=1966828, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.010999] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:24.011113] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=109][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774203846405249, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774204010986) [2024-04-22 16:23:24.011144] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774203810941, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:24.017050] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.017157] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=6000, remain_us=1957766, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.027007] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.027118] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=7000, remain_us=1947806, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.027616] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=77][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:320, tid:10796}]) [2024-04-22 16:23:24.034267] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=37][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:7, local_retry_times:7, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.034338] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=54][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.034368] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.034379] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.034422] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.036964] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=51][errcode=-4725] [2024-04-22 16:23:24.036996] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.037026] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.037056] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.037075] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.037084] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.037093] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.037130] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.037179] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.037203] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=21][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.037219] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=8, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.037247] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15] will sleep(sleep_us=8000, remain_us=1937677, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.045396] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:8, local_retry_times:8, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.045509] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=94][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.045554] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=42][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.045572] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.045594] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.049144] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:24.049188] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=42][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.049208] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.049227] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.049255] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.049285] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.049304] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.049345] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.049401] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.049493] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=89][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.049527] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=9, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.049572] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=31] will sleep(sleep_us=9000, remain_us=1925352, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.058722] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:9, local_retry_times:9, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.058790] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=50][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.058812] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.058828] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.058867] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.060181] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:519) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=4679] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2024-04-22 16:23:24.061787] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] [2024-04-22 16:23:24.061859] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=69][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.061905] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=43][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.061920] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.061938] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.061967] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.061983] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.062034] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.062094] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=53][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.062109] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.062130] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=10, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.062175] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=31] will sleep(sleep_us=10000, remain_us=1912748, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.064739] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=40] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:24.066368] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:161, local_retry_times:161, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.066497] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.066554] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=53][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.066570] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.066595] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.068417] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:534) [11430][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=24] worker thread exit(tenant_->id()=1, workers_.get_size()=1) [2024-04-22 16:23:24.068530] INFO [SHARE] end_run (ob_tenant_base.cpp:332) [11430][T1_L0_G1][T0][Y0-0000000000000000-0-0] [lt=101] tenant thread end_run(id_=1, ret=0, thread_count_=225) [2024-04-22 16:23:24.069400] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=56][errcode=-4725] [2024-04-22 16:23:24.069481] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=79][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.069537] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=52][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.069561] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.069583] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.069594] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.069605] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.069663] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.069766] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.069790] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:24.069827] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=162, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.069899] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55] will sleep(sleep_us=100000, remain_us=10757244, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=162, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.072335] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:10, local_retry_times:10, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.072468] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=78][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.072491] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.072503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.072527] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.076040] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:24.076087] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.076118] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.076136] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.076149] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.076161] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.076171] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.076217] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.076289] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=64][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.076311] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.076331] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=11, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.076364] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18] will sleep(sleep_us=11000, remain_us=1898559, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.087508] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:11, local_retry_times:11, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.087587] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=57][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.087614] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.087627] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.087652] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.091310] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:24.091346] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=34][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.091393] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=44][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.091414] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.091431] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.091483] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=50][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.091494] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.091540] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.091576] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.091589] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.091606] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=12, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.091638] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18] will sleep(sleep_us=12000, remain_us=1883286, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.103785] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:12, local_retry_times:12, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.103885] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=79][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.103914] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.103926] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.103962] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.107411] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=34][errcode=-4725] [2024-04-22 16:23:24.107485] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=71][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.107513] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.107530] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.107543] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.107554] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.107564] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.107608] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.107680] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=64][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.107702] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.107721] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=13, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.107755] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19] will sleep(sleep_us=13000, remain_us=1867168, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.109119] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=76] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:24.113398] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=67] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:24.120963] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=40][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:13, local_retry_times:13, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.121060] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=76][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.121082] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.121093] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.121117] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.124565] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:24.124648] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=80][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.124680] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.124698] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.124712] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.124723] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.124734] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.124779] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.124816] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.124828] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.124845] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=14, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.124878] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18] will sleep(sleep_us=14000, remain_us=1850045, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.139060] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=34][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:14, local_retry_times:14, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.139144] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.139172] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.139192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.139277] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.142851] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] [2024-04-22 16:23:24.142809] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [10835][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=30] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f3cec9dc2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-22 16:23:24.142890] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.142922] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.142947] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.142967] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.142984] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.143000] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.143059] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.143137] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=70][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.143159] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.143179] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=15, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.143210] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18] will sleep(sleep_us=15000, remain_us=1831713, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.158364] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:15, local_retry_times:15, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.158474] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=57][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.158495] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.158506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.158535] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.161326] INFO [SHARE] pre_run (ob_tenant_base.cpp:314) [11497][][T1][Y0-0000000000000000-0-0] [lt=0] tenant thread pre_run(MTL_ID()=1, ret=0, thread_count_=226) [2024-04-22 16:23:24.161331] INFO [SERVER.OMT] recv_group_request (ob_tenant.cpp:1418) [10461][pnio1][T0][YB42AC175882-000616AB0B5C3C7A-0-0] [lt=17] worker thread created(id()=1, group->group_id_=1) [2024-04-22 16:23:24.161476] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [11497][][T1][Y0-0000000000000000-0-0] [lt=85] Init thread local success [2024-04-22 16:23:24.161878] INFO [OCCAM] get_idx (ob_occam_time_guard.h:223) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C7A-0-0] [lt=27] init point thread id with(&point=0x5596475a9b40, idx_=3305, point=[thread id=11497, timeout ts=08:00:00.0, last click point="(null):(null):0", last click ts=08:00:00.0], thread_id=11497) [2024-04-22 16:23:24.162008] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] [2024-04-22 16:23:24.162029] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.162050] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.162065] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.162078] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.162089] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.162099] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.162145] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.162180] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.162212] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.162235] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=21][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=16, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.162277] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25] will sleep(sleep_us=16000, remain_us=1812646, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.168703] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF288-0-0] [lt=79] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=32, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:24.168757] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF288-0-0] [lt=51] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=33925, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:24.168767] INFO [PALF] read (log_cache.cpp:100) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=854, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:24.168782] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF288-0-0] [lt=23] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=23, avg_log_submit_to_flush_time=4940, avg_log_submit_to_slide_time=33902) [2024-04-22 16:23:24.168920] INFO [LIB] stat (utility.h:1165) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=24] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=97, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:24.168981] INFO [LIB] stat (utility.h:1165) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=59] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=11, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:24.170049] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:162, local_retry_times:162, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.170139] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=59][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.170161] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.170172] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.170197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.173657] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:24.173700] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.173755] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.173797] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.173811] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.173822] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.173833] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.173881] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.173928] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.173944] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:24.173961] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=163, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.173993] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=10653149, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=163, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.178413] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:16, local_retry_times:16, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.178514] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=80][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.178539] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.178578] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=37][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.178621] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.182426] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] [2024-04-22 16:23:24.182540] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=111][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.182576] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=32][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.182597] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.182614] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.182629] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.182643] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.182701] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.182761] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=52][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.182789] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=24][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.182847] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=54][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=17, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.182891] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29] will sleep(sleep_us=17000, remain_us=1792032, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.190497] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.190508] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.190591] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=70] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.190895] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=88] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.190899] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.190940] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=52] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.191728] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=70] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.191737] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.191762] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.192091] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.192919] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.192936] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=48] gc stale ls task succ [2024-04-22 16:23:24.193085] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.193268] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.193379] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.193516] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.194077] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.194283] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=24] ====== check clog disk timer task ====== [2024-04-22 16:23:24.194337] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=44] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:24.194357] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.194401] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=32] cannot_recycle_log_size statistics(cannot_recycle_log_size=365692, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:24.194405] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:24.194472] INFO pn_ratelimit (group.c:589) [10608][IngressService][T0][Y0-0000000000000000-0-0] [lt=42] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:23:24.194492] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:24.194505] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10922, mem_used=79030272) [2024-04-22 16:23:24.194634] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.195593] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.195919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=88] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.196234] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.196483] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=20] current monitor number(seq_=-1) [2024-04-22 16:23:24.197166] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.197527] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=89] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.198705] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:24.198865] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=24][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:24.198941] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:24.199011] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=65][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:23:24.199034] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=21][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:24.199053] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:23:24.199072] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:24.199086] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:24.199103] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:23:24.199116] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:24.199127] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:24.199138] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:24.199149] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:23:24.199160] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:23:24.199170] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=9][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:23:24.199190] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=18][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:24.199203] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:24.199215] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:24.199226] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:24.199251] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:24.199264] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:24.199279] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:24.199291] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:24.199302] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:23:24.199317] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:24.199329] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.199391] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=56][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:24.199420] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:24.199431] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:24.199482] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=49][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:24.199514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC53-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:24.199534] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10899][BlackListServic][T0][YB42AC175881-000616AB07BFEC53-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:23:24.199549] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:24.199564] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:24.199583] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:24.199597] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1498050, start=1713774204198538, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:24.199616] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:24.199630] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:24.199645] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13] start to print blacklist info [2024-04-22 16:23:24.199735] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11] ls blacklist refresh finish(cost_time=2791) [2024-04-22 16:23:24.199926] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=5][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:24.200013] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=69] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2024-04-22 16:23:24.200043] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2024-04-22 16:23:24.200061] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2024-04-22 16:23:24.200080] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2024-04-22 16:23:24.200097] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:24.200110] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:24.200126] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2024-04-22 16:23:24.200139] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:24.200163] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=22][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:24.200141] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:17, local_retry_times:17, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.200174] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:24.200186] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:24.200197] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:24.200200] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=41][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.200209] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:24.200218] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.200221] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:24.200229] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.200244] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:24.200257] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:24.200271] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:24.200254] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.200290] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=17][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:24.200302] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", ret=-5019) [2024-04-22 16:23:24.200316] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:24.200328] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.200352] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:24.200376] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:24.200387] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:24.200396] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:24.200417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:24.200433] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10814][T1_DupTbLease][T1][YB42AC175881-000616AB0A5FEBD7-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, aret=-5019, ret=-5019) [2024-04-22 16:23:24.200497] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:24.200513] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:24.200525] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:24.200539] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cbe430050, start=1713774204199330, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:24.200556] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:24.200569] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:943) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:23:24.200660] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:672) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2024-04-22 16:23:24.200680] WDIAG [STORAGE.DUP_TABLE] refresh_dup_tablet_schema_ (ob_dup_table_util.cpp:97) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] get duplicate ls status info failed(ret=-5019, dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2024-04-22 16:23:24.200709] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:148) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-5019] refresh dup table schema failed(ret=-5019) [2024-04-22 16:23:24.200721] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:236) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] scan dup ls to find dup_tablet failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:24.200743] WDIAG [STORAGE.DUP_TABLE] runTimerTask (ob_dup_table_util.cpp:73) [10814][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] execute dup ls scan failed(tmp_ret=-5019) [2024-04-22 16:23:24.201526] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:24.201578] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:24.201601] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:24.201619] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:24.203781] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=41][errcode=-4725] [2024-04-22 16:23:24.203836] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=52][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:24.203876] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=36][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:24.203893] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:24.203907] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:24.203918] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:24.203928] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.203989] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.204065] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=64][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:24.204097] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:24.204136] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=36][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=18, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.204234] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=68] will sleep(sleep_us=18000, remain_us=1770690, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.211269] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:24.211417] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=102][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774203956521477, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774204211251) [2024-04-22 16:23:24.211541] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=121][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774204011187, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:24.215833] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [10520][AutoLSLocRpc][T0][YB42AC175881-000616AB0A9FED7A-0-0] [lt=87] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, location:{server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:8}}]) [2024-04-22 16:23:24.216987] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:24.217050] WDIAG [SERVER] do_heartbeat_event (ob_heartbeat.cpp:266) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] try reload schema failed(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=-4023) [2024-04-22 16:23:24.222278] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=79] PNIO [ratelimit] time: 1713774204222275, bytes: 66442158, bw: 0.030484 MB/s, add_ts: 1000021, add_bytes: 31965 [2024-04-22 16:23:24.222505] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=52][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:18, local_retry_times:18, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:24.222600] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=76][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.222622] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:24.222632] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:24.222656] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:24.226077] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.226162] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=19000, remain_us=1748762, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.229098] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=25][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:24.229174] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:24.229212] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=33][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:24.229235] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=21][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:24.229254] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:24.229273] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:24.229285] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:24.229302] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:24.229314] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:24.229325] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:24.229336] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:24.229347] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:24.229367] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=18][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:24.229380] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:24.229391] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:24.229414] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:24.229427] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:24.229480] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=49][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:24.229493] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:24.229504] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:24.229519] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:24.229531] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.229563] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=26][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:24.229591] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:24.229615] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:24.229626] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:24.229648] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8B-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:24.229666] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:24.229682] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:24.229696] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:24.229707] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:24.229720] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cf3bc0050, start=1713774204228714, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:24.229856] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=134][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:24.229879] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=19][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:24.229992] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=31][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:24.230023] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8B-0-0] [lt=29][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:24.230349] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=49] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713774203729363], gts:1713774203727113067, latest_srr:[mts=1713774204230257]}) [2024-04-22 16:23:24.240505] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C1-0-0] [lt=273][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:24.240566] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C1-0-0] [lt=57][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:24.240615] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C1-0-0] [lt=45][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:24.240657] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C1-0-0] [lt=41][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:24.247943] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=20] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:24.248980] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.249088] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1725836, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.250192] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=178][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:24.250280] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=66] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:24.250322] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=38][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:24.250349] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=25][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:24.250378] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=27][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:24.250405] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=20][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:24.250425] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=18][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:24.250500] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=64][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:24.250520] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=17][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:24.250537] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=16][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:24.250577] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=38][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:24.250589] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:24.250600] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:24.250612] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:24.250623] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:24.250647] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:24.250660] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:24.250675] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:24.250686] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:24.250698] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:24.250712] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:24.250724] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:24.250762] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=31][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:24.250798] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=28][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:24.250810] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:24.250819] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:24.250852] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:24.250868] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:24.250883] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:24.250897] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:24.250909] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:24.250922] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f3d1177a050, start=1713774204249974, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:24.250939] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:24.250959] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=17][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:24.251049] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=15][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:24.251093] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=40][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:24.251117] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=22][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:24.251133] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=15][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:24.251148] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:24.251163] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=12][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:24.251175] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED87-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:24.264928] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:24.265842] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0D1FEC93-0-0] [lt=63][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:24.273291] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.273378] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=21000, remain_us=1701545, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.277832] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=56] PNIO [ratelimit] time: 1713774204277828, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007590, add_bytes: 0 [2024-04-22 16:23:24.277932] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.277983] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10549159, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=164, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.299901] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.299989] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=22000, remain_us=1674934, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.314229] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=109] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:24.316672] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316732] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316760] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316799] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316813] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316826] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316840] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316853] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316865] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316878] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316892] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316905] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316925] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316939] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316952] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.316965] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:24.325130] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.325169] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=23000, remain_us=1649754, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.338814] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=115] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:24.353604] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.353687] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1621236, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.382931] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.383015] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=1591908, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.383122] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.383226] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=10443916, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=165, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.411520] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:24.411589] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774204254472426, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774204411506) [2024-04-22 16:23:24.411642] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774204211566, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:24.411708] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=156202, log_service_ts=1713774204254472, min_tx_service_ts=4611686018427387, timestamp={val:1713774204254472426, v:0}) [2024-04-22 16:23:24.411737] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=100, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713774204254472426, v:0}", last_lsn={lsn:365748}, last_gts={val:1713774187494514341, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:23:24.412041] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.412124] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=26000, remain_us=1562799, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.441929] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.442016] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=27000, remain_us=1532908, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.465140] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=58] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:24.468819] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1172) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=40] succ to submit log task to replay service(to_submit_lsn={lsn:365861}, to_submit_scn={val:1713774204452623409, v:0}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774204468718, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:365927}, next_to_submit_scn_:{val:1713774204452623410, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:365805}, end_lsn:{lsn:365927}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774204452623409, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774204452623409, v:0}, data_checksum:2923900223, flag:1}}, init_mode_version:0, accumulate_checksum:2780365498, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774204452623409, v:0}}}}}) [2024-04-22 16:23:24.469207] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=35][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:24.472749] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.472832] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=28000, remain_us=1502091, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.487614] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.487719] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10339424, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=166, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.504701] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.504838] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=29000, remain_us=1470086, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.514945] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=63] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:24.537798] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.537908] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=30000, remain_us=1437016, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.561483] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C80-0-0] [lt=25] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:24.561570] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C80-0-0] [lt=54] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=845, avg_group_log_size=120) [2024-04-22 16:23:24.561603] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C80-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=845) [2024-04-22 16:23:24.571869] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.571989] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=31000, remain_us=1402935, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.587877] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:24.591960] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.592055] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=10235088, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=167, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.605388] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.605451] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=32000, remain_us=1369472, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.611731] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:24.611801] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774204452623409, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774204611718) [2024-04-22 16:23:24.611831] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774204411664, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:24.639791] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.639841] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=33000, remain_us=1335082, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.639871] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFDCC-0-0] [lt=113][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774204639776}) [2024-04-22 16:23:24.665354] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=68] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:24.676993] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.677072] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1297851, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.696300] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.696427] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=10130717, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=168, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.711844] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:24.715029] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.715122] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1259802, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.715675] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:24.731886] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=75] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:24.733645] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C4-0-0] [lt=196][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:24.733711] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C4-0-0] [lt=62][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:24.733807] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C4-0-0] [lt=91][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:24.733864] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C4-0-0] [lt=54][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:24.754080] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.754169] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=36000, remain_us=1220755, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.794127] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.794221] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=37000, remain_us=1180703, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.800367] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.800500] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10026642, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=169, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.811940] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:24.811983] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:24.812039] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774204553563846, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774204811931) [2024-04-22 16:23:24.812067] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774204611871, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:24.835367] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.835509] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=38000, remain_us=1139415, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.865584] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=69] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:24.867734] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=967) [2024-04-22 16:23:24.876355] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.876497] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=39000, remain_us=1098426, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.904420] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.904553] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=9922590, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=170, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:24.916318] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=81] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:24.918965] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.919051] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=1055872, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.962754] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:24.962903] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=41000, remain_us=1012020, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:24.968119] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=64] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=5498, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:24.968212] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=65] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=70, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:24.968370] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=88, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:25.005784] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [10532][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=49] runTimer to refresh locality_info(ret=0) [2024-04-22 16:23:25.008000] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.008040] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=50][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:25.008119] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=42000, remain_us=966805, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.008142] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=83] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:25.008174] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:25.008194] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:25.008237] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=41][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:25.008256] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:25.008237] INFO [CLOG] operator() (ob_log_replay_service.cpp:1492) [10817][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=33] get_replay_process success(id={id:1}, replayed_log_size=366537, unreplayed_log_size=0) [2024-04-22 16:23:25.008269] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:25.008286] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:23:25.008299] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:25.008310] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:25.008321] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:25.008332] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=9][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:23:25.008328] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:156) [10817][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=37] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-04-22 16:23:25.008342] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=9][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:23:25.008353] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=10][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:23:25.008364] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:25.008376] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:25.008388] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:25.008413] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=24][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:25.008483] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=18][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:25.008507] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=20][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.008529] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.008547] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=16][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:25.008564] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=15][errcode=-5019] fail to handle text query(stmt=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name, ret=-5019) [2024-04-22 16:23:25.008586] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=19][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:25.008605] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=17][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.008641] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=30][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:25.008677] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=29][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.008713] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=34][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.008689] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.008724] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=11][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:25.008757] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=9818386, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=171, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.008757] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10531][LocltyRefTask][T1][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:25.008787] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10531][LocltyRefTask][T0][YB42AC175881-000616AB0AEFEBDF-0-0] [lt=25][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, aret=-5019, ret=-5019) [2024-04-22 16:23:25.008812] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-04-22 16:23:25.008833] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:25.008852] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:25.008887] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1484050, start=1713774205007676, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-04-22 16:23:25.008915] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:25.008935] WDIAG [SHARE] load_region (ob_locality_table_operator.cpp:158) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-04-22 16:23:25.008956] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=20] load region(ret=-5019, locality_info={version:0, local_region:"", local_zone:"", local_idc:"", local_zone_type:3, local_zone_status:3, locality_region_array:[], locality_zone_array:[]}) [2024-04-22 16:23:25.009098] WDIAG [STORAGE] load_region (ob_locality_manager.cpp:236) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-5019] localitity operator load region error(ret=-5019) [2024-04-22 16:23:25.009159] WDIAG [STORAGE] process (ob_locality_manager.cpp:762) [10531][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=51][errcode=-5019] process refresh locality task fail(ret=-5019) [2024-04-22 16:23:25.010283] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [10521][DumpLSLoc][T0][YB42AC175881-000616AB0AFFEBDF-0-0] [lt=110] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, renew_time:1713774202196081, replica_locations:[{server:"172.23.88.130:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:8}, {server:"172.23.88.131:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.23.88.129:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2024-04-22 16:23:25.012194] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:25.012278] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=79][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774204855825210, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774205012181) [2024-04-22 16:23:25.012325] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774204812089, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:25.054123] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.054212] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=43000, remain_us=920711, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.065812] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:25.066998] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=11][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:25.067066] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=49] table not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, table_name.ptr()="data_size:25, data:5F5F616C6C5F6469736B5F696F5F63616C6962726174696F6E", ret=-5019) [2024-04-22 16:23:25.067096] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, ret=-5019) [2024-04-22 16:23:25.067113] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_disk_io_calibration, db_name=oceanbase) [2024-04-22 16:23:25.067130] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_disk_io_calibration) [2024-04-22 16:23:25.067166] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=31][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:25.067178] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:25.067193] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_disk_io_calibration' doesn't exist [2024-04-22 16:23:25.067205] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:25.067215] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:25.067225] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=9][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:25.067234] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:25.067245] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:25.067256] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:25.067266] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:25.067287] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:25.067300] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.067326] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=22][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.067337] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:25.067347] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA", ret=-5019) [2024-04-22 16:23:25.067360] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:25.067371] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.067395] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:25.067419] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.067429] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.067489] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=58][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:25.067521] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC46-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA""}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:25.067539] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10404][ServerGTimer][T0][YB42AC175881-000616AB07DFEC46-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA""}, aret=-5019, ret=-5019) [2024-04-22 16:23:25.067560] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:23:25.067574] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:25.067585] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:25.067597] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd2268050, start=1713774205066785, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:23:25.067613] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:25.067624] WDIAG [COMMON] parse_calibration_table (ob_io_calibration.cpp:838) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, sql_string=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.129" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:23:25.067712] WDIAG [COMMON] read_from_table (ob_io_calibration.cpp:708) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] parse calibration data failed(ret=-5019) [2024-04-22 16:23:25.067725] WDIAG [SERVER] refresh_io_calibration (ob_server.cpp:3426) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] fail to refresh io calibration from table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.067736] WDIAG [SERVER] runTimerTask (ob_server.cpp:3416) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] ObRefreshIOCalibrationTimeTask task failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.101241] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.101333] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=44000, remain_us=873591, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.110298] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=59] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:25.111037] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED46-0-0] [lt=28] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:25.111725] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.111812] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=9715330, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=172, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.116850] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=92] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:25.117279] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=12][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:25.117355] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=58] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-04-22 16:23:25.117390] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-04-22 16:23:25.117417] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=25][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-04-22 16:23:25.117496] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=75][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-04-22 16:23:25.117526] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=21][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:25.117545] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=18][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:25.117563] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-04-22 16:23:25.117575] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:25.117587] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:25.117598] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:25.117609] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:25.117675] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=63][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:25.117696] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=18][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:25.117710] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=12][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:25.117762] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:25.117776] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.117791] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.117803] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:25.117814] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-04-22 16:23:25.117829] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:25.117841] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.117867] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:25.117910] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=36][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.117921] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.117931] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:25.117963] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:25.117982] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:25.117998] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:23:25.118012] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:25.118024] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:25.118037] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd148e050, start=1713774205117071, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:23:25.118055] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:25.118068] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1177) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=10][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-04-22 16:23:25.118321] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:872) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=20][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:23:25.118363] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:68) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=38][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.118387] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=21][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.118401] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:94) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=13][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.118414] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:108) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=12] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.120094] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=16][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:25.120162] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-04-22 16:23:25.120190] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-04-22 16:23:25.120208] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-04-22 16:23:25.120225] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-04-22 16:23:25.120242] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:25.120279] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10678][OmtNodeBalancer][T1][YB42AC175881-000616AB06EFEBE2-0-0] [lt=35][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:25.120613] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:301) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=1] cal_all_part_disk_default_percentage succ(data_dir="/home/creality/oceanbase/store/sstable", clog_dir="/home/creality/oceanbase/store/clog", shared_mode=false, data_disk_total_size=205110317056, data_disk_default_percentage=90, clog_disk_total_size=85856358400, clog_disk_default_percentage=90) [2024-04-22 16:23:25.120684] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:337) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=68] decide disk size finished(suggested_disk_size=81604378624, suggested_disk_percentage=0, default_disk_percentage=90, total_space=85856358400, disk_size=81604378624) [2024-04-22 16:23:25.120715] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=28] get_log_disk_info_in_config(suggested_data_disk_size=161061273600, suggested_clog_disk_size=81604378624, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=81604378624, log_disk_percentage=0, total_log_disk_size=85856358400) [2024-04-22 16:23:25.120738] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=18] try_resize success(ret=0, log_disk_size=81604378624, total_log_disk_size=85856358400, this={dir::"/home/creality/oceanbase/store/clog/log_pool", dir_fd:14, meta_fd:15, log_pool_meta:{curr_total_size:81604378624, next_total_size:81604378624, status:0}, min_block_id:2, max_block_id:1216, min_log_disk_size_for_all_tenants_:4294967296, is_inited:true}) [2024-04-22 16:23:25.120767] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:132) [10678][OmtNodeBalancer][T0][YB42AC175881-000616AB06EFEBE2-0-0] [lt=28] refresh tenant config(tenants=[], ret=-5019) [2024-04-22 16:23:25.130677] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=58][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:520, tid:10477}]) [2024-04-22 16:23:25.134257] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=12] === Run print tenant memory usage task === [2024-04-22 16:23:25.134409] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=93] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 581,132,288 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 192,438,272 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 115,343,360 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 33,783,808 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 201,011,200 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:25.134202] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=40] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:3, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1713773681580810, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=9, offset=1469}, ls_meta_entry:[5](ver=0,mode=0,seq=4), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:53, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:233666747, recv_hp_rpc_cnt:1, recv_np_rpc_cnt:958, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:1409344, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=15443 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:2, queue_size:0, recv_req_cnt:2158, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773684733335}{group_id:10, queue_size:0, recv_req_cnt:1375, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773689875802}{group_id:5, queue_size:0, recv_req_cnt:813, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773687232900}{group_id:19, queue_size:0, recv_req_cnt:40, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1713774180915685}{group_id:9, queue_size:0, recv_req_cnt:12374, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773677273264}{group_id:1, queue_size:0, recv_req_cnt:5826, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1713774204160585}, rpc_stat_info: pcode=0x14a:cnt=273 pcode=0x1501:cnt=54 pcode=0x1516:cnt=41 pcode=0x4a9:cnt=30 pcode=0x150a:cnt=20, token_change_ts:1713773687300994, tenant_role:1}) [2024-04-22 16:23:25.135503] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=104] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 192,438,272 used= 181,752,112 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,477,760 used= 31,458,304 count= 1 avg_used= 31,458,304 block_cnt= 1 chunk_cnt= 1 mod=ash_list [MEMORY] hold= 22,761,472 used= 22,699,200 count= 9 avg_used= 2,522,133 block_cnt= 9 chunk_cnt= 8 mod=IoControl [MEMORY] hold= 16,809,984 used= 15,702,816 count= 228 avg_used= 68,872 block_cnt= 228 chunk_cnt= 30 mod=[T]ObSessionDIB [MEMORY] hold= 12,873,728 used= 12,856,352 count= 1 avg_used= 12,856,352 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 9,457,664 used= 9,437,784 count= 1 avg_used= 9,437,784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckInteChe [MEMORY] hold= 5,513,216 used= 5,502,336 count= 21 avg_used= 262,016 block_cnt= 21 chunk_cnt= 9 mod=ObTxDesc [MEMORY] hold= 5,348,288 used= 5,251,272 count= 50 avg_used= 105,025 block_cnt= 26 chunk_cnt= 12 mod=PartitTableTask [MEMORY] hold= 5,267,456 used= 5,190,496 count= 8 avg_used= 648,812 block_cnt= 8 chunk_cnt= 3 mod=KvstCachWashStr [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBucTenComMo [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=PxP2PDhMgrKey [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashPxBlooFilKe [MEMORY] hold= 4,307,328 used= 4,278,711 count= 3 avg_used= 1,426,237 block_cnt= 3 chunk_cnt= 3 mod=SqlDtlMgr [MEMORY] hold= 4,231,936 used= 4,203,008 count= 6 avg_used= 700,501 block_cnt= 5 chunk_cnt= 3 mod=BaseLogWriter [MEMORY] hold= 4,214,880 used= 4,194,328 count= 2 avg_used= 2,097,164 block_cnt= 2 chunk_cnt= 2 mod=SerFuncRegHT [MEMORY] hold= 3,276,800 used= 3,214,848 count= 11 avg_used= 292,258 block_cnt= 11 chunk_cnt= 7 mod=DedupQueue [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdUnitMa [MEMORY] hold= 2,167,040 used= 2,129,912 count= 7 avg_used= 304,273 block_cnt= 7 chunk_cnt= 5 mod=FixedQueue [MEMORY] hold= 2,114,080 used= 2,031,616 count= 1,026 avg_used= 1,980 block_cnt= 209 chunk_cnt= 3 mod=TabletLSMap [MEMORY] hold= 2,037,344 used= 1,915,008 count= 468 avg_used= 4,091 block_cnt= 243 chunk_cnt= 3 mod=CharsetInit [MEMORY] hold= 1,843,200 used= 1,638,400 count= 25 avg_used= 65,536 block_cnt= 25 chunk_cnt= 14 mod=[T]char [MEMORY] hold= 1,591,232 used= 1,560,960 count= 21 avg_used= 74,331 block_cnt= 20 chunk_cnt= 9 mod=TenantCtxAlloca [MEMORY] hold= 1,581,056 used= 1,572,904 count= 1 avg_used= 1,572,904 block_cnt= 1 chunk_cnt= 1 mod=IdConnMap [MEMORY] hold= 1,523,712 used= 1,507,118 count= 5 avg_used= 301,423 block_cnt= 5 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 1,460,608 used= 1,442,816 count= 45 avg_used= 32,062 block_cnt= 45 chunk_cnt= 21 mod=CommonArray [MEMORY] hold= 1,408,800 used= 1,388,000 count= 4 avg_used= 347,000 block_cnt= 4 chunk_cnt= 3 mod=Omt [MEMORY] hold= 1,286,144 used= 1,270,296 count= 2 avg_used= 635,148 block_cnt= 2 chunk_cnt= 2 mod=PxResMgr [MEMORY] hold= 1,142,912 used= 1,107,072 count= 558 avg_used= 1,984 block_cnt= 147 chunk_cnt= 24 mod=CreateContext [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=ConcurHashMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=TsSourceInfoMap [MEMORY] hold= 933,888 used= 897,024 count= 5 avg_used= 179,404 block_cnt= 5 chunk_cnt= 5 mod=LightyQueue [MEMORY] hold= 933,888 used= 917,576 count= 2 avg_used= 458,788 block_cnt= 2 chunk_cnt= 1 mod=CACHE_INST_MAP [MEMORY] hold= 827,472 used= 773,160 count= 13 avg_used= 59,473 block_cnt= 12 chunk_cnt= 8 mod=HashBucket [MEMORY] hold= 704,512 used= 691,200 count= 2 avg_used= 345,600 block_cnt= 2 chunk_cnt= 2 mod=LDBlockBitMap [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=GEleTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=EventTimer [MEMORY] hold= 622,592 used= 619,008 count= 1 avg_used= 619,008 block_cnt= 1 chunk_cnt= 1 mod=SysTaskStatus [MEMORY] hold= 614,400 used= 556,560 count= 15 avg_used= 37,104 block_cnt= 15 chunk_cnt= 8 mod=InneSqlConnPool [MEMORY] hold= 573,440 used= 304,920 count= 35 avg_used= 8,712 block_cnt= 35 chunk_cnt= 13 mod=[T]ObDSActionAr [MEMORY] hold= 512,000 used= 507,904 count= 64 avg_used= 7,936 block_cnt= 64 chunk_cnt= 2 mod=ModulePageAlloc [MEMORY] hold= 491,264 used= 441,808 count= 271 avg_used= 1,630 block_cnt= 116 chunk_cnt= 13 mod=tg [MEMORY] hold= 458,752 used= 457,856 count= 7 avg_used= 65,408 block_cnt= 7 chunk_cnt= 6 mod=ObTsResponseTas [MEMORY] hold= 448,000 used= 444,416 count= 56 avg_used= 7,936 block_cnt= 56 chunk_cnt= 13 mod=SessionInfoHash [MEMORY] hold= 425,984 used= 420,032 count= 1 avg_used= 420,032 block_cnt= 1 chunk_cnt= 1 mod=CompEventMgr [MEMORY] hold= 403,648 used= 378,880 count= 258 avg_used= 1,468 block_cnt= 54 chunk_cnt= 2 mod=LSLocationMap [MEMORY] hold= 401,408 used= 393,256 count= 1 avg_used= 393,256 block_cnt= 1 chunk_cnt= 1 mod=TablStorStatMgr [MEMORY] hold= 368,896 used= 330,944 count= 7 avg_used= 47,277 block_cnt= 7 chunk_cnt= 3 mod=Rpc [MEMORY] hold= 327,680 used= 325,960 count= 5 avg_used= 65,192 block_cnt= 5 chunk_cnt= 5 mod=LatchStat [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=register_task [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=register_tasks [MEMORY] hold= 262,144 used= 261,632 count= 4 avg_used= 65,408 block_cnt= 4 chunk_cnt= 4 mod=AsyncTaskQueue [MEMORY] hold= 229,376 used= 224,000 count= 1 avg_used= 224,000 block_cnt= 1 chunk_cnt= 1 mod=BGTMonitor [MEMORY] hold= 221,184 used= 214,432 count= 1 avg_used= 214,432 block_cnt= 1 chunk_cnt= 1 mod=CompSuggestMgr [MEMORY] hold= 221,184 used= 212,992 count= 1 avg_used= 212,992 block_cnt= 1 chunk_cnt= 1 mod=TSWorker [MEMORY] hold= 212,992 used= 204,928 count= 1 avg_used= 204,928 block_cnt= 1 chunk_cnt= 1 mod=TenantMutilAllo [MEMORY] hold= 212,992 used= 196,624 count= 2 avg_used= 98,312 block_cnt= 2 chunk_cnt= 2 mod=DdlQue [MEMORY] hold= 212,992 used= 196,624 count= 2 avg_used= 98,312 block_cnt= 2 chunk_cnt= 1 mod=DRTaskMap [MEMORY] hold= 211,744 used= 210,008 count= 27 avg_used= 7,778 block_cnt= 27 chunk_cnt= 8 mod=HashNode [MEMORY] hold= 180,224 used= 172,064 count= 1 avg_used= 172,064 block_cnt= 1 chunk_cnt= 1 mod=TenantMBList [MEMORY] hold= 163,840 used= 147,792 count= 2 avg_used= 73,896 block_cnt= 2 chunk_cnt= 1 mod=HashNodNexWaiMa [MEMORY] hold= 163,504 used= 155,416 count= 2 avg_used= 77,708 block_cnt= 2 chunk_cnt= 2 mod=OB_DISK_REP [MEMORY] hold= 160,624 used= 159,272 count= 21 avg_used= 7,584 block_cnt= 21 chunk_cnt= 2 mod=TenaSpaTabIdSet [MEMORY] hold= 155,648 used= 153,152 count= 3 avg_used= 51,050 block_cnt= 3 chunk_cnt= 2 mod=IO_MGR [MEMORY] hold= 155,648 used= 147,624 count= 1 avg_used= 147,624 block_cnt= 1 chunk_cnt= 1 mod=UsrRuleMap [MEMORY] hold= 144,880 used= 143,656 count= 19 avg_used= 7,560 block_cnt= 19 chunk_cnt= 2 mod=SysTableNameMap [MEMORY] hold= 139,264 used= 131,776 count= 1 avg_used= 131,776 block_cnt= 1 chunk_cnt= 1 mod=GtsTaskQueue [MEMORY] hold= 137,920 used= 134,912 count= 11 avg_used= 12,264 block_cnt= 11 chunk_cnt= 3 mod=SeArray [MEMORY] hold= 124,416 used= 122,688 count= 27 avg_used= 4,544 block_cnt= 27 chunk_cnt= 11 mod=[T]ObTraceEvent [MEMORY] hold= 122,880 used= 120,408 count= 3 avg_used= 40,136 block_cnt= 3 chunk_cnt= 3 mod=CommSysVarFac [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=RefrFullScheMap [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=MemMgrMap [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=MemMgrForLiboMa [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=TenaSchForCacMa [MEMORY] hold= 115,616 used= 114,656 count= 15 avg_used= 7,643 block_cnt= 15 chunk_cnt= 2 mod=HashNodeConfCon [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=NonPartTenMap [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=IndNameMap [MEMORY] hold= 114,336 used= 106,088 count= 2 avg_used= 53,044 block_cnt= 2 chunk_cnt= 2 mod=RetryCtrl [MEMORY] hold= 106,496 used= 98,312 count= 1 avg_used= 98,312 block_cnt= 1 chunk_cnt= 1 mod=TmpFileManager [MEMORY] hold= 106,496 used= 86,408 count= 5 avg_used= 17,281 block_cnt= 5 chunk_cnt= 2 mod=HashBuckConfCon [MEMORY] hold= 88,000 used= 56,704 count= 10 avg_used= 5,670 block_cnt= 9 chunk_cnt= 5 mod=BucketLock [MEMORY] hold= 81,920 used= 74,064 count= 2 avg_used= 37,032 block_cnt= 2 chunk_cnt= 2 mod=io_trace_map [MEMORY] hold= 79,328 used= 70,816 count= 8 avg_used= 8,852 block_cnt= 8 chunk_cnt= 7 mod=MallocInfoMap [MEMORY] hold= 73,728 used= 65,600 count= 1 avg_used= 65,600 block_cnt= 1 chunk_cnt= 1 mod=TCREF [MEMORY] hold= 73,728 used= 69,664 count= 1 avg_used= 69,664 block_cnt= 1 chunk_cnt= 1 mod=SuperBlockBuffe [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=ObLSLocation [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=KVCACHE_HAZARD [MEMORY] hold= 65,136 used= 63,096 count= 2 avg_used= 31,548 block_cnt= 2 chunk_cnt= 2 mod=ScheCacSysCacMa [MEMORY] hold= 64,000 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 7 mod=RsEventQueue [MEMORY] hold= 64,000 used= 63,488 count= 8 avg_used= 7,936 block_cnt= 8 chunk_cnt= 5 mod=SqlSession [MEMORY] hold= 57,344 used= 55,368 count= 1 avg_used= 55,368 block_cnt= 1 chunk_cnt= 1 mod=DepInfoTaskQ [MEMORY] hold= 56,128 used= 27,136 count= 377 avg_used= 71 block_cnt= 222 chunk_cnt= 22 mod=Coro [MEMORY] hold= 49,152 used= 32,768 count= 2 avg_used= 16,384 block_cnt= 2 chunk_cnt= 1 mod=CACHE_TNT_LST [MEMORY] hold= 49,152 used= 37,032 count= 3 avg_used= 12,344 block_cnt= 3 chunk_cnt= 2 mod=ReferedMap [MEMORY] hold= 49,152 used= 36,912 count= 2 avg_used= 18,456 block_cnt= 2 chunk_cnt= 2 mod=HashBuckSysConf [MEMORY] hold= 46,912 used= 37,632 count= 18 avg_used= 2,090 block_cnt= 14 chunk_cnt= 5 mod=TGTimer [MEMORY] hold= 45,760 used= 45,056 count= 11 avg_used= 4,096 block_cnt= 11 chunk_cnt= 6 mod=LinearHashMapCn [MEMORY] hold= 45,760 used= 45,056 count= 11 avg_used= 4,096 block_cnt= 11 chunk_cnt= 6 mod=LinearHashMapDi [MEMORY] hold= 41,264 used= 37,264 count= 2 avg_used= 18,632 block_cnt= 2 chunk_cnt= 1 mod=TaskRunnerSer [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucConPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=T3MTBMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=ClientSessBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=KillSessMapBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdConfMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucNamConMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DDLSpeedCtrl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HasBucConRefCoM [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TmpFileStoreMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdPoolMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=ObLongopsMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucNamPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucSerUniMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucTenPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HasBucSerMigUnM [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SessHoldMapBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SqlLoadData [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucPooUniMa [MEMORY] hold= 29,696 used= 26,864 count= 39 avg_used= 688 block_cnt= 29 chunk_cnt= 15 mod=ObGuard [MEMORY] hold= 24,944 used= 20,784 count= 2 avg_used= 10,392 block_cnt= 2 chunk_cnt= 2 mod=SqlNio [MEMORY] hold= 24,672 used= 24,264 count= 6 avg_used= 4,044 block_cnt= 6 chunk_cnt= 2 mod=ScheObSchemAren [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=GrpIdNameMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=ServerCkptSlogH [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=leakMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=GrpNameIdMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=FuncRuleMap [MEMORY] hold= 24,224 used= 21,672 count= 2 avg_used= 10,836 block_cnt= 2 chunk_cnt= 1 mod=SchemaStatuMap [MEMORY] hold= 24,000 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 3 mod=PlanVaIdx [MEMORY] hold= 23,808 used= 5,448 count= 227 avg_used= 24 block_cnt= 141 chunk_cnt= 23 mod=[T]MemoryContex [MEMORY] hold= 21,504 used= 20,352 count= 16 avg_used= 1,272 block_cnt= 16 chunk_cnt= 2 mod=IO_GROUP_MAP [MEMORY] hold= 18,416 used= 18,144 count= 4 avg_used= 4,536 block_cnt= 4 chunk_cnt= 3 mod=DeviceMng [MEMORY] hold= 16,384 used= 12,296 count= 1 avg_used= 12,296 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 16,384 used= 10,592 count= 1 avg_used= 10,592 block_cnt= 1 chunk_cnt= 1 mod=TenCompProgMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=GenSchemVersMap [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=LinkArray [MEMORY] hold= 16,384 used= 8,992 count= 1 avg_used= 8,992 block_cnt= 1 chunk_cnt= 1 mod=IO_HEALTH [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TbltRefreshMap [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ServerLogPool [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MemDumpMap [MEMORY] hold= 16,384 used= 9,336 count= 1 avg_used= 9,336 block_cnt= 1 chunk_cnt= 1 mod=InnerLobHash [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=HasBucTimZonInM [MEMORY] hold= 16,032 used= 15,904 count= 2 avg_used= 7,952 block_cnt= 2 chunk_cnt= 1 mod=UpgProcSet [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=SqlSessiVarMap [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=CommSysVarDefVa [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=MacroIdList [MEMORY] hold= 11,712 used= 8,944 count= 43 avg_used= 208 block_cnt= 26 chunk_cnt= 4 mod=Scheduler [MEMORY] hold= 10,816 used= 9,984 count= 13 avg_used= 768 block_cnt= 13 chunk_cnt= 4 mod=timer [MEMORY] hold= 9,472 used= 8,384 count= 16 avg_used= 524 block_cnt= 4 chunk_cnt= 2 mod=RpcBuffer [MEMORY] hold= 8,976 used= 6,696 count= 31 avg_used= 216 block_cnt= 13 chunk_cnt= 11 mod=ObFuture [MEMORY] hold= 8,384 used= 8,104 count= 4 avg_used= 2,026 block_cnt= 3 chunk_cnt= 2 mod=TenantInfo [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=acceptOK [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PrepareOk [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=AddrList [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SchemaRowKey [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerCidMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerIdcMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=followerRenew [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ObTsTenantInfoN [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=Node [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RetryTask [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerBlacklist [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerRegioMap [MEMORY] hold= 7,968 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=LuaHandler [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HasNodTzInfM [MEMORY] hold= 6,704 used= 3,840 count= 40 avg_used= 96 block_cnt= 30 chunk_cnt= 9 mod=PThread [MEMORY] hold= 5,872 used= 4,424 count= 20 avg_used= 221 block_cnt= 16 chunk_cnt= 11 mod=EventReHolder [MEMORY] hold= 5,424 used= 5,288 count= 2 avg_used= 2,644 block_cnt= 2 chunk_cnt= 2 mod=DeadLock [MEMORY] hold= 5,376 used= 5,248 count= 2 avg_used= 2,624 block_cnt= 1 chunk_cnt= 1 mod=RootContext [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=SqlPx [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmReq [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=RebuildCtx [MEMORY] hold= 3,888 used= 3,816 count= 1 avg_used= 3,816 block_cnt= 1 chunk_cnt= 1 mod=RemMasterMap [MEMORY] hold= 3,888 used= 3,816 count= 1 avg_used= 3,816 block_cnt= 1 chunk_cnt= 1 mod=RecScheHisMap [MEMORY] hold= 3,680 used= 3,360 count= 5 avg_used= 672 block_cnt= 5 chunk_cnt= 4 mod=[T]ObWarningBuf [MEMORY] hold= 3,264 used= 3,200 count= 1 avg_used= 3,200 block_cnt= 1 chunk_cnt= 1 mod=TenantTZ [MEMORY] hold= 3,184 used= 1,728 count= 20 avg_used= 86 block_cnt= 19 chunk_cnt= 11 mod=ObFunction [MEMORY] hold= 2,768 used= 2,704 count= 1 avg_used= 2,704 block_cnt= 1 chunk_cnt= 1 mod=LoggerAlloc [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=StorageS3 [MEMORY] hold= 1,728 used= 1,536 count= 3 avg_used= 512 block_cnt= 3 chunk_cnt= 2 mod=TntResourceMgr [MEMORY] hold= 1,600 used= 1,536 count= 1 avg_used= 1,536 block_cnt= 1 chunk_cnt= 1 mod=LobManager [MEMORY] hold= 1,504 used= 1,432 count= 1 avg_used= 1,432 block_cnt= 1 chunk_cnt= 1 mod=GtsRequestRpc [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=T3MQSyncLock [MEMORY] hold= 1,472 used= 1,400 count= 1 avg_used= 1,400 block_cnt= 1 chunk_cnt= 1 mod=GtiRequestRpc [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=TsMgr [MEMORY] hold= 1,408 used= 1,344 count= 1 avg_used= 1,344 block_cnt= 1 chunk_cnt= 1 mod=SchemaService [MEMORY] hold= 1,392 used= 1,328 count= 1 avg_used= 1,328 block_cnt= 1 chunk_cnt= 1 mod=GtsRpcProxy [MEMORY] hold= 1,392 used= 1,328 count= 1 avg_used= 1,328 block_cnt= 1 chunk_cnt= 1 mod=GtiRpcProxy [MEMORY] hold= 1,360 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=TENANT_PLAN_MAP [MEMORY] hold= 1,360 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=INGRESS_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IO_CHANNEL_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 1,184 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=memdumpqueue [MEMORY] hold= 1,168 used= 1,092 count= 1 avg_used= 1,092 block_cnt= 1 chunk_cnt= 1 mod=LogRegionMap [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 2 mod=SqlSessiQuerSql [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 2 mod=SqlString [MEMORY] hold= 1,008 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=PackStateMap [MEMORY] hold= 992 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 1 mod=SequenceIdMap [MEMORY] hold= 992 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=ContextsMap [MEMORY] hold= 992 used= 848 count= 2 avg_used= 424 block_cnt= 2 chunk_cnt= 2 mod=SequenceMap [MEMORY] hold= 768 used= 704 count= 1 avg_used= 704 block_cnt= 1 chunk_cnt= 1 mod=ScheMgrCacheMap [MEMORY] hold= 640 used= 280 count= 5 avg_used= 56 block_cnt= 5 chunk_cnt= 1 mod=Log [MEMORY] hold= 528 used= 264 count= 3 avg_used= 88 block_cnt= 3 chunk_cnt= 2 mod=DestKAState [MEMORY] hold= 432 used= 272 count= 2 avg_used= 136 block_cnt= 2 chunk_cnt= 2 mod=unknown [MEMORY] hold= 288 used= 84 count= 3 avg_used= 28 block_cnt= 2 chunk_cnt= 2 mod=KeepAliveServer [MEMORY] hold= 240 used= 48 count= 3 avg_used= 16 block_cnt= 3 chunk_cnt= 3 mod=CreateEntity [MEMORY] hold= 224 used= 160 count= 1 avg_used= 160 block_cnt= 1 chunk_cnt= 1 mod=OccamTimeGuard [MEMORY] hold= 128 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=PxTargetMgr [MEMORY] hold= 80 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=Compressor [MEMORY] hold= 80 used= 7 count= 1 avg_used= 7 block_cnt= 1 chunk_cnt= 1 mod=SqlExpr [MEMORY] hold= 181,752,112 used= 178,329,612 count= 4,501 avg_used= 39,619 mod=SUMMARY [2024-04-22 16:23:25.135900] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=283] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 6,291,456 used= 2,657,680 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,600,016 used= 675,052 count= 12,703 avg_used= 53 block_cnt= 209 chunk_cnt= 2 mod=Buffer [MEMORY] hold= 1,004,144 used= 703,383 count= 3,304 avg_used= 212 block_cnt= 200 chunk_cnt= 3 mod=glibc_malloc [MEMORY] hold= 53,520 used= 37,729 count= 229 avg_used= 164 block_cnt= 29 chunk_cnt= 1 mod=S3SDK [MEMORY] hold= 2,657,680 used= 1,416,164 count= 16,236 avg_used= 87 mod=SUMMARY [2024-04-22 16:23:25.135987] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=54] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 115,343,360 used= 109,412,352 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 109,412,352 used= 109,208,832 count= 212 avg_used= 515,136 block_cnt= 212 chunk_cnt= 55 mod=CoStack [MEMORY] hold= 109,412,352 used= 109,208,832 count= 212 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:25.136046] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=34] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 6,291,456 used= 3,882,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,882,960 used= 3,662,624 count= 265 avg_used= 13,821 block_cnt= 47 chunk_cnt= 3 mod=OB_TEST2_PCODE [MEMORY] hold= 3,882,960 used= 3,662,624 count= 265 avg_used= 13,821 mod=SUMMARY [2024-04-22 16:23:25.136123] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=34] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 14,680,064 used= 14,565,376 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10,403,840 used= 10,398,720 count= 5 avg_used= 2,079,744 block_cnt= 5 chunk_cnt= 5 mod=Logger [MEMORY] hold= 4,161,536 used= 4,159,488 count= 2 avg_used= 2,079,744 block_cnt= 2 chunk_cnt= 2 mod=ErrorLogger [MEMORY] hold= 14,565,376 used= 14,558,208 count= 7 avg_used= 2,079,744 mod=SUMMARY [2024-04-22 16:23:25.135328] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1122] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:5, max_cpu:5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713773672949917, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"5.000000000000000000e+00", unit_max_cpu:"5.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:52, max_worker_cnt:150, stopped:0, worker_us:93310959, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2367148, workers:52, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1713773687322359, tenant_role:0}) [2024-04-22 16:23:25.136343] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=63] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 33,783,808 used= 31,774,480 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 25,499,408 used= 25,349,144 count= 52 avg_used= 487,483 block_cnt= 23 chunk_cnt= 12 mod=DEFAULT [MEMORY] hold= 2,752,512 used= 2,750,496 count= 21 avg_used= 130,976 block_cnt= 21 chunk_cnt= 4 mod=PKTC_INBUF [MEMORY] hold= 2,752,512 used= 2,750,496 count= 21 avg_used= 130,976 block_cnt= 21 chunk_cnt= 4 mod=PKTS_INBUF [MEMORY] hold= 245,760 used= 244,320 count= 15 avg_used= 16,288 block_cnt= 15 chunk_cnt= 4 mod=SERVER_CTX_CHUN [MEMORY] hold= 196,608 used= 195,456 count= 12 avg_used= 16,288 block_cnt= 12 chunk_cnt= 3 mod=SERVER_RESP_CHU [MEMORY] hold= 196,608 used= 195,456 count= 12 avg_used= 16,288 block_cnt= 12 chunk_cnt= 3 mod=CLIENT_CB_CHUNK [MEMORY] hold= 131,072 used= 130,304 count= 8 avg_used= 16,288 block_cnt= 8 chunk_cnt= 2 mod=CLIENT_REQ_CHUN [MEMORY] hold= 31,774,480 used= 31,615,672 count= 141 avg_used= 224,224 mod=SUMMARY [2024-04-22 16:23:25.136845] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=57] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 11,292,672 used= 9,760,320 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 7,098,368 used= 7,078,824 count= 1 avg_used= 7,078,824 block_cnt= 1 chunk_cnt= 1 mod=SchemaIdVersion [MEMORY] hold= 2,088,768 used= 2,087,680 count= 2 avg_used= 1,043,840 block_cnt= 2 chunk_cnt= 2 mod=TenantSchemMgr [MEMORY] hold= 360,448 used= 327,680 count= 4 avg_used= 81,920 block_cnt= 4 chunk_cnt= 1 mod=SchemaMgrCache [MEMORY] hold= 110,208 used= 102,397 count= 5 avg_used= 20,479 block_cnt= 3 chunk_cnt= 1 mod=SchemaSysCache [MEMORY] hold= 31,872 used= 31,616 count= 4 avg_used= 7,904 block_cnt= 4 chunk_cnt= 1 mod=ScheTenaInfoVec [MEMORY] hold= 16,320 used= 16,064 count= 4 avg_used= 4,016 block_cnt= 4 chunk_cnt= 1 mod=SchemaSysVariab [MEMORY] hold= 15,936 used= 15,808 count= 2 avg_used= 7,904 block_cnt= 2 chunk_cnt= 1 mod=ScheTablInfoVec [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheIndeNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheTablIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=HiddenTblNames [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheRoutIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheRoutNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheTablNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=SchePackIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=SchePackNameMap [MEMORY] hold= 1,536 used= 1,024 count= 8 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSeCompo [MEMORY] hold= 1,152 used= 768 count= 6 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSePolic [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=ScheUdtNameMap [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=ScheUdtIdMap [MEMORY] hold= 1,152 used= 768 count= 6 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSeLabel [MEMORY] hold= 800 used= 512 count= 4 avg_used= 128 block_cnt= 3 chunk_cnt= 1 mod=SchemaSynonym [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=DBLINK_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabSeUserLe [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 4 chunk_cnt= 1 mod=DIRECTORY_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_POLICY_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlSqlMap [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_GROUP_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaProfile [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_CONTEXT_MGR [MEMORY] hold= 656 used= 584 count= 1 avg_used= 584 block_cnt= 1 chunk_cnt= 1 mod=TenSchMemMgrFoL [MEMORY] hold= 656 used= 584 count= 1 avg_used= 584 block_cnt= 1 chunk_cnt= 1 mod=TenaScheMemMgr [MEMORY] hold= 416 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheDataNameMap [MEMORY] hold= 416 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=MockFkParentTab [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaSecurAudi [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheAuxVpNameVe [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheForKeyNamMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaContext [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheConsNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlIdMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchePriTabPriMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchePriObjPriMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheTrigIdMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaTablespac [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheTrigNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaUdf [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaSequence [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaKeystore [MEMORY] hold= 9,760,320 used= 9,691,189 count= 134 avg_used= 72,322 mod=SUMMARY [2024-04-22 16:23:25.137053] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=126] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 201,011,200 used= 199,007,152 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 134,238,288 used= 134,217,736 count= 2 avg_used= 67,108,868 block_cnt= 2 chunk_cnt= 2 mod=CACHE_MAP_BKT [MEMORY] hold= 18,051,072 used= 18,033,840 count= 1 avg_used= 18,033,840 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MB_HANDLE [MEMORY] hold= 17,329,216 used= 8,968,960 count= 1,093 avg_used= 8,205 block_cnt= 1,093 chunk_cnt= 9 mod=StorageLoggerM [MEMORY] hold= 16,806,784 used= 16,786,208 count= 3 avg_used= 5,595,402 block_cnt= 3 chunk_cnt= 2 mod=FixeSizeBlocAll [MEMORY] hold= 6,311,936 used= 6,291,472 count= 1 avg_used= 6,291,472 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MAP_LOCK [MEMORY] hold= 3,712,256 used= 3,298,048 count= 122 avg_used= 27,033 block_cnt= 55 chunk_cnt= 4 mod=OccamThreadPool [MEMORY] hold= 2,543,616 used= 2,523,256 count= 1 avg_used= 2,523,256 block_cnt= 1 chunk_cnt= 1 mod=TenantConfig [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=BlockMap [MEMORY] hold= 5,984 used= 1,088 count= 68 avg_used= 16 block_cnt= 2 chunk_cnt= 2 mod=ConfigChecker [MEMORY] hold= 199,007,152 used= 190,128,544 count= 1,292 avg_used= 147,158 mod=SUMMARY [2024-04-22 16:23:25.137154] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=53] [MEMORY] tenant: 508, limit: 1,073,741,824 hold: 40,427,520 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 8,970,240 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 31,457,280 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:25.136213] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [10677][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=881] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:2.5, max_cpu:2.5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713773672930258, is_removed:false}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"2.500000000000000000e+00", unit_max_cpu:"2.500000000000000000e+00", total_worker_cnt:35, min_worker_cnt:27, max_worker_cnt:150, stopped:0, worker_us:70361341, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:1183000, workers:27, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1713773687342553, tenant_role:0}) [2024-04-22 16:23:25.137235] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=30] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 8,970,240 used= 7,859,136 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 4,423,680 used= 4,132,320 count= 60 avg_used= 68,872 block_cnt= 60 chunk_cnt= 3 mod=[T]ObSessionDIB [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 28,800 used= 24,960 count= 60 avg_used= 416 block_cnt= 7 chunk_cnt= 2 mod=OMT_Worker [MEMORY] hold= 16,320 used= 7,680 count= 120 avg_used= 64 block_cnt= 7 chunk_cnt= 2 mod=Coro [MEMORY] hold= 5,888 used= 1,440 count= 60 avg_used= 24 block_cnt= 7 chunk_cnt= 2 mod=[T]MemoryContex [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 7,859,136 used= 7,522,400 count= 304 avg_used= 24,744 mod=SUMMARY [2024-04-22 16:23:25.137382] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=49] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 31,457,280 used= 30,965,760 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 30,965,760 used= 30,908,160 count= 60 avg_used= 515,136 block_cnt= 60 chunk_cnt= 15 mod=CoStack [MEMORY] hold= 30,965,760 used= 30,908,160 count= 60 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:25.137714] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=41] [MEMORY] tenant: 509, limit: 1,073,741,824 hold: 25,747,456 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 6,873,088 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 18,874,368 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:25.137855] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=73] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 6,873,088 used= 5,994,736 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,580,480 used= 2,410,520 count= 35 avg_used= 68,872 block_cnt= 35 chunk_cnt= 2 mod=[T]ObSessionDIB [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 16,800 used= 14,560 count= 35 avg_used= 416 block_cnt= 4 chunk_cnt= 1 mod=OMT_Worker [MEMORY] hold= 9,648 used= 4,480 count= 70 avg_used= 64 block_cnt= 4 chunk_cnt= 1 mod=Coro [MEMORY] hold= 3,360 used= 840 count= 35 avg_used= 24 block_cnt= 4 chunk_cnt= 1 mod=[T]MemoryContex [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 5,994,736 used= 5,786,400 count= 179 avg_used= 32,326 mod=SUMMARY [2024-04-22 16:23:25.138036] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=53] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 18,874,368 used= 18,063,360 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 18,063,360 used= 18,029,760 count= 35 avg_used= 515,136 block_cnt= 35 chunk_cnt= 9 mod=CoStack [MEMORY] hold= 18,063,360 used= 18,029,760 count= 35 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:25.138305] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=37] [MEMORY] tenant: 1, limit: 4,294,967,296 hold: 648,929,280 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 453,873,664 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 130,023,424 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 2,097,152 limit= 858,993,440 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 18,894,848 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33,554,432 limit= 9,223,372,036,854,775,807 [2024-04-22 16:23:25.139403] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=34] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 453,873,664 used= 381,360,768 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 31 [MEMORY] hold= 159,105,024 used= 159,041,216 count= 40 avg_used= 3,976,030 block_cnt= 40 chunk_cnt= 40 mod=MysqlRequesReco [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 23,678,976 used= 23,614,912 count= 23 avg_used= 1,026,735 block_cnt= 23 chunk_cnt= 12 mod=IoControl [MEMORY] hold= 16,662,528 used= 15,565,072 count= 226 avg_used= 68,872 block_cnt= 226 chunk_cnt= 32 mod=[T]ObSessionDIB [MEMORY] hold= 14,483,248 used= 14,360,745 count= 93 avg_used= 154,416 block_cnt= 33 chunk_cnt= 7 mod=OmtTenant [MEMORY] hold= 8,986,624 used= 8,945,664 count= 2 avg_used= 4,472,832 block_cnt= 2 chunk_cnt= 2 mod=ServerObjecPool [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,257,536 used= 8,240,064 count= 2 avg_used= 4,120,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 5,234,688 used= 4,653,056 count= 71 avg_used= 65,536 block_cnt= 71 chunk_cnt= 9 mod=[T]char [MEMORY] hold= 4,943,872 used= 4,915,456 count= 2 avg_used= 2,457,728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4,571,136 used= 4,525,056 count= 6 avg_used= 754,176 block_cnt= 6 chunk_cnt= 5 mod=ResourceGroup [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 4,214,784 used= 4,194,336 count= 1 avg_used= 4,194,336 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 4 mod=SqlDtlQueue [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,864 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,166,784 used= 2,142,208 count= 2 avg_used= 1,071,104 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 2,080,768 used= 2,079,744 count= 1 avg_used= 2,079,744 block_cnt= 1 chunk_cnt= 1 mod=LogReplayTask [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 2 mod=DagMap [MEMORY] hold= 1,777,664 used= 1,770,696 count= 1 avg_used= 1,770,696 block_cnt= 1 chunk_cnt= 1 mod=TableProc [MEMORY] hold= 1,596,912 used= 1,588,632 count= 3 avg_used= 529,544 block_cnt= 3 chunk_cnt= 2 mod=TabletMap [MEMORY] hold= 1,584,288 used= 1,575,992 count= 3 avg_used= 525,330 block_cnt= 2 chunk_cnt= 1 mod=PartitTableTask [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1,269,184 used= 1,263,904 count= 6 avg_used= 210,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 786,560 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 3 mod=LogAlloc [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 655,360 used= 650,752 count= 1 avg_used= 650,752 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 446,976 used= 440,768 count= 97 avg_used= 4,544 block_cnt= 97 chunk_cnt= 17 mod=[T]ObTraceEvent [MEMORY] hold= 442,368 used= 438,432 count= 3 avg_used= 146,144 block_cnt= 3 chunk_cnt= 2 mod=SQLSessionInfo [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 403,648 used= 378,880 count= 258 avg_used= 1,468 block_cnt= 53 chunk_cnt= 3 mod=LSMap [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 385,024 used= 377,120 count= 1 avg_used= 377,120 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 365,040 used= 362,160 count= 45 avg_used= 8,048 block_cnt= 45 chunk_cnt= 11 mod=[T]ObPerfEventR [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompStMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=RSCompactMap [MEMORY] hold= 212,992 used= 212,160 count= 1 avg_used= 212,160 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 196,608 used= 196,224 count= 3 avg_used= 65,408 block_cnt= 3 chunk_cnt= 3 mod=StorMetaCacheIO [MEMORY] hold= 152,000 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 9 mod=LSIter [MEMORY] hold= 147,456 used= 78,408 count= 9 avg_used= 8,712 block_cnt= 9 chunk_cnt= 6 mod=[T]ObDSActionAr [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 130,944 used= 130,816 count= 2 avg_used= 65,408 block_cnt= 2 chunk_cnt= 2 mod=ResultSet [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 100,008 count= 1 avg_used= 100,008 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 98,304 used= 97,408 count= 1 avg_used= 97,408 block_cnt= 1 chunk_cnt= 1 mod=tenant_mf_mgr [MEMORY] hold= 98,304 used= 96,032 count= 1 avg_used= 96,032 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 94,240 used= 90,112 count= 64 avg_used= 1,408 block_cnt= 17 chunk_cnt= 5 mod=TxCtxMgr [MEMORY] hold= 81,920 used= 65,536 count= 2 avg_used= 32,768 block_cnt= 2 chunk_cnt= 1 mod=CsDictPl [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 72,240 used= 1,024 count= 896 avg_used= 1 block_cnt= 10 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 64,352 used= 31,248 count= 453 avg_used= 68 block_cnt= 123 chunk_cnt= 32 mod=Coro [MEMORY] hold= 56,448 used= 55,584 count= 12 avg_used= 4,632 block_cnt= 12 chunk_cnt= 5 mod=WorkerMap [MEMORY] hold= 49,152 used= 48,576 count= 1 avg_used= 48,576 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 48,832 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=RawPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=DictPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=CsIntPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=CsStrPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=ConstPl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40,000 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 36,064 used= 32,928 count= 49 avg_used= 672 block_cnt= 20 chunk_cnt= 8 mod=[T]ObWarningBuf [MEMORY] hold= 32,768 used= 17,408 count= 2 avg_used= 8,704 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTTP [MEMORY] hold= 32,768 used= 25,600 count= 1 avg_used= 25,600 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_ClientTask [MEMORY] hold= 32,576 used= 26,079 count= 2 avg_used= 13,039 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 25,440 used= 22,048 count= 53 avg_used= 416 block_cnt= 14 chunk_cnt= 6 mod=OMT_Worker [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=RlePl [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=IntDiffPl [MEMORY] hold= 22,160 used= 5,424 count= 226 avg_used= 24 block_cnt= 91 chunk_cnt= 26 mod=[T]MemoryContex [MEMORY] hold= 17,472 used= 16,128 count= 21 avg_used= 768 block_cnt= 18 chunk_cnt= 6 mod=TGTimer [MEMORY] hold= 16,800 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 3 chunk_cnt= 3 mod=BaseLogWriter [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_TableCtxMgr [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=HexStrPl [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 13,696 count= 1 avg_used= 13,696 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ColEqualPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=StrDiffPl [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxIMgr [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 16,384 used= 12,288 count= 1 avg_used= 12,288 block_cnt= 1 chunk_cnt= 1 mod=DiskIteratorSto [MEMORY] hold= 15,744 used= 15,616 count= 2 avg_used= 7,808 block_cnt= 2 chunk_cnt= 2 mod=HashNodeLCSta [MEMORY] hold= 8,320 used= 8,192 count= 2 avg_used= 4,096 block_cnt= 2 chunk_cnt= 2 mod=LinearHashMapCn [MEMORY] hold= 8,320 used= 8,192 count= 2 avg_used= 4,096 block_cnt= 2 chunk_cnt= 2 mod=LinearHashMapDi [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TruncateLog [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTableMgr [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LSInfo [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=TabletIDSetNode [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePlanCac [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoNode [MEMORY] hold= 7,856 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 7,856 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletNode [MEMORY] hold= 5,184 used= 5,120 count= 1 avg_used= 5,120 block_cnt= 1 chunk_cnt= 1 mod=CtxBlockPl [MEMORY] hold= 4,800 used= 4,656 count= 2 avg_used= 2,328 block_cnt= 2 chunk_cnt= 2 mod=CaLsInfoMap [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,160 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4,160 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=CsCtxBlockPl [MEMORY] hold= 3,072 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,336 used= 1,728 count= 8 avg_used= 216 block_cnt= 4 chunk_cnt= 3 mod=LogIOTask [MEMORY] hold= 1,984 used= 1,920 count= 1 avg_used= 1,920 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 1,872 used= 1,296 count= 9 avg_used= 144 block_cnt= 5 chunk_cnt= 4 mod=ObTxLSLogCb [MEMORY] hold= 1,872 used= 1,800 count= 1 avg_used= 1,800 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,664 used= 1,600 count= 1 avg_used= 1,600 block_cnt= 1 chunk_cnt= 1 mod=LogPartFetCtxPo [MEMORY] hold= 1,616 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,488 used= 1,352 count= 2 avg_used= 676 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=LockMemtableMgr [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,184 used= 1,056 count= 2 avg_used= 528 block_cnt= 1 chunk_cnt= 1 mod=PoolArenaArray [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,120 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 848 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=TLD_TIMER [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=CheckPointTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=FlushTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=DiskUsageTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=TabShellTimer [MEMORY] hold= 720 used= 648 count= 1 avg_used= 648 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 624 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 544 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 480 used= 120 count= 5 avg_used= 24 block_cnt= 3 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 336 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 192 used= 102 count= 1 avg_used= 102 block_cnt= 1 chunk_cnt= 1 mod=TxLSLogBuf [MEMORY] hold= 192 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 176 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 176 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 160 used= 16 count= 2 avg_used= 8 block_cnt= 2 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 128 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 112 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 80 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 381,360,768 used= 375,132,571 count= 28,355 avg_used= 13,229 mod=SUMMARY [2024-04-22 16:23:25.139828] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=373] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 2,097,152 used= 24,000 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 24,000 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 1 mod=SqlExecContext [MEMORY] hold= 24,000 used= 23,808 count= 3 avg_used= 7,936 mod=SUMMARY [2024-04-22 16:23:25.139865] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 794,624 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 794,624 used= 788,032 count= 1 avg_used= 788,032 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 794,624 used= 788,032 count= 1 avg_used= 788,032 mod=SUMMARY [2024-04-22 16:23:25.139908] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 2,097,152 used= 1,286,528 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,232,192 used= 1,115,344 count= 43 avg_used= 25,938 block_cnt= 43 chunk_cnt= 1 mod=SqlPhyPlan [MEMORY] hold= 32,000 used= 31,744 count= 4 avg_used= 7,936 block_cnt= 4 chunk_cnt= 1 mod=SqlPlanCache [MEMORY] hold= 20,608 used= 19,840 count= 10 avg_used= 1,984 block_cnt= 3 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 1,728 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 1,286,528 used= 1,168,528 count= 59 avg_used= 19,805 mod=SUMMARY [2024-04-22 16:23:25.139988] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=32] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,848 used= 59,018 count= 263 avg_used= 224 block_cnt= 20 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,944 used= 59,050 count= 264 avg_used= 223 mod=SUMMARY [2024-04-22 16:23:25.140075] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=59] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 130,023,424 used= 125,927,424 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 125,927,424 used= 125,693,184 count= 244 avg_used= 515,136 block_cnt= 244 chunk_cnt= 62 mod=CoStack [MEMORY] hold= 125,927,424 used= 125,693,184 count= 244 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:23:25.140181] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=32] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 2,097,152 used= 466,944 limit= 858,993,440 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 401,408 used= 400,064 count= 2 avg_used= 200,032 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=N_TabletPool [MEMORY] hold= 466,944 used= 465,472 count= 3 avg_used= 155,157 mod=SUMMARY [2024-04-22 16:23:25.140264] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=45] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 65,536 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=LobAllocator [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 mod=SUMMARY [2024-04-22 16:23:25.140357] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=45] [MEMORY] tenant_id= 1 ctx_id= RPC_CTX_ID hold= 0 used= 16,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16,384 used= 16,256 count= 1 avg_used= 16,256 block_cnt= 1 chunk_cnt= 1 mod=OB_REMOTE_EXECU [MEMORY] hold= 16,384 used= 16,256 count= 1 avg_used= 16,256 mod=SUMMARY [2024-04-22 16:23:25.140495] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=47] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 18,894,848 used= 17,870,464 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17,862,464 used= 17,833,792 count= 3 avg_used= 5,944,597 block_cnt= 3 chunk_cnt= 2 mod=MEMTABLE_ARENA [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 17,870,464 used= 17,841,728 count= 4 avg_used= 4,460,432 mod=SUMMARY [2024-04-22 16:23:25.140638] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [10409][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=74] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33,554,432 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 33,554,432 free_size= 33,554,432 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2024-04-22 16:23:25.140773] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:122) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=32] ====== tenants memory info ====== === TENANTS MEMORY INFO === divisive_memory_used= 85,721,088 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 581,132,288 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 40,427,520 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 25,747,456 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,713,774,205,138,267 active_memstore_used= 0 total_memstore_used= 0 total_memstore_hold= 0 memstore_freeze_trigger_limit= 343,597,360 memstore_limit= 1,717,986,880 mem_tenant_limit= 4,294,967,296 mem_tenant_hold= 648,929,280 max_mem_memstore_can_get_now= 1,823,019,000 memstore_alloc_pos= 0 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2024-04-22 16:23:25.141183] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:162) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=46] [CHUNK_MGR] free=14 pushes=63159 pops=63145 limit= 26,843,545,600 hold= 1,329,823,744 total_hold= 1,421,869,056 used= 1,296,236,544 freelist_hold= 33,587,200 large_freelist_hold= 4,227,072 maps= 434 unmaps= 12 large_maps= 34 large_unmaps= 5 huge_maps= 32 huge_unmaps= 3 memalign=0 resident_size= 1,769,037,824 virtual_memory_used= 2,297,397,248 [2024-04-22 16:23:25.141275] INFO print (ob_malloc_time_monitor.cpp:39) [10409][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=47] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 0, 10): delta_total_cost_time= 2392052, delta_count= 17662327, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10, 100): delta_total_cost_time= 25997, delta_count= 1845, avg_cost_time= 14 [MALLOC_TIME_MONITOR] [ 100, 1000): delta_total_cost_time= 170, delta_count= 1, avg_cost_time= 170 [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2024-04-22 16:23:25.145504] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=54][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:44, local_retry_times:44, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.145582] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=58][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.145611] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.145629] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.145726] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=64][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.149273] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=38][errcode=-4725] [2024-04-22 16:23:25.149326] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=51][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.149359] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.149382] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.149398] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.149410] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.149421] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.149522] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.149560] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.149574] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.149591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=45, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.149647] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=39] will sleep(sleep_us=45000, remain_us=825276, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.186410] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:588) [10807][T1_TntSharedTim][T1][Y0-0000000000000000-0-0] [lt=55] dump dtl interm result cost(us)(dump_cost=4377, ret=0, interm count=0, dump count=52) [2024-04-22 16:23:25.190148] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:603) [10807][T1_TntSharedTim][T1][Y0-0000000000000000-0-0] [lt=95] clear dtl interm result cost(us)(clear_cost=3640, ret=0, gc_.expire_keys_.count()=0, dump count=52, interm count=0, clean count=0) [2024-04-22 16:23:25.190412] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.190429] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=32] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.190639] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=55] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.190704] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.191064] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.191061] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=47] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.191090] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.191296] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=100] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.191510] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.191850] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.192540] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.192566] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=43] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.192603] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:222) [10510][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=33] [ASYNC TASK QUEUE](queue_size=0) [2024-04-22 16:23:25.192948] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.193096] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=34] gc stale ls task succ [2024-04-22 16:23:25.193133] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.193172] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=55] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.194011] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=83] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.194583] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=68] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:25.194679] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:25.194730] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.194799] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=53] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=10967, mem_used=79030272) [2024-04-22 16:23:25.194771] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:45, local_retry_times:45, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.194873] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=80][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.194908] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=31][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.194973] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=62][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.195012] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=23][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.195346] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.195351] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.195622] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.197336] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=54] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.197531] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.198270] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=57] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.198746] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=70][errcode=-4725] [2024-04-22 16:23:25.198786] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.198815] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.198831] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.198845] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.198856] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.198867] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.198922] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.198994] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=62][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.199038] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=40][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.199066] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=46, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.199100] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19] will sleep(sleep_us=46000, remain_us=775824, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.199096] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:25.202096] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:25.202156] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:25.202183] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:25.202222] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:25.211400] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=50] ====== tenant freeze timer task ====== [2024-04-22 16:23:25.212001] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=67][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:172, local_retry_times:172, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.212111] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=79][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.212133] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.212144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.212169] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.212474] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:25.212542] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774205058408833, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774205212424) [2024-04-22 16:23:25.212614] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774205012352, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:25.216157] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=63][errcode=-4725] [2024-04-22 16:23:25.216203] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=43][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.216234] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.216253] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.216267] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.216278] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.216289] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.216344] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.216421] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=66][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.216506] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=80][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:25.216536] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=173, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.216598] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46] will sleep(sleep_us=100000, remain_us=9610545, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=173, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.229270] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=147] PNIO [ratelimit] time: 1713774205229266, bytes: 66471885, bw: 0.028153 MB/s, add_ts: 1006991, add_bytes: 29727 [2024-04-22 16:23:25.231052] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=79][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:31, tid:10410}]) [2024-04-22 16:23:25.232053] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=17][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:25.232127] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=55] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:25.232159] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:25.232178] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:25.232196] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:25.232214] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:25.232227] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:25.232244] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:25.232257] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:25.232268] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:25.232279] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:25.232312] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=31][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:25.232325] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:25.232337] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:25.232348] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:25.232371] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:25.232385] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.232399] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.232411] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:25.232423] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:25.232484] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=57][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:25.232506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=19][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.232532] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:25.232575] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=36][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.232586] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.232596] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:25.232627] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8C-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:25.232648] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:25.232664] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:25.232678] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:25.232690] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:25.232703] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd227a050, start=1713774205231830, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:25.232721] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:25.232742] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=17][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:25.232834] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=14][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:25.232846] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8C-0-0] [lt=11][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:25.238574] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C7-0-0] [lt=287][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:25.238656] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C7-0-0] [lt=75][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:25.238721] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C7-0-0] [lt=61][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:25.238812] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35C7-0-0] [lt=86][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:25.245276] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:46, local_retry_times:46, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.245393] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.245415] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.245426] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.245569] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=100][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.249174] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=73][errcode=-4725] [2024-04-22 16:23:25.249214] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.249257] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=40][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.249273] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.249287] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.249299] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.249310] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.249357] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.249411] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.249427] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.249489] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=59][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=47, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.249525] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20] will sleep(sleep_us=47000, remain_us=725399, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.252320] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:25.254368] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=224][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:25.254468] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=49] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:25.254503] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:25.254523] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:25.254542] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:25.254560] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:25.254573] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:25.254610] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=28][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:25.254623] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:25.254634] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:25.254645] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:25.254690] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:25.254705] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=14][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:25.254718] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:25.254730] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:25.254786] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:25.254802] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.254817] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:25.254828] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:25.254853] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=22][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:25.254868] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:25.254880] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.254905] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:25.254931] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.254942] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:25.254952] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:25.254976] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:25.255000] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=22][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:25.255020] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=18][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:25.255042] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=20][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:25.255074] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=29][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:25.255103] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=26][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45cc050, start=1713774205254161, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:25.255189] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=83][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:25.255216] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=22][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:25.255317] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=18][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:25.255349] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=29][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:25.255416] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=64][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.255514] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=95][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.255541] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=21][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:25.255567] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=21][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:25.255590] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED88-0-0] [lt=21][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:25.266032] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=68] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:25.268488] INFO [PALF] read (log_cache.cpp:100) [10868][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1089, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:25.268382] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF291-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=33, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:25.268638] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF291-0-0] [lt=249] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=17069, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:25.268660] INFO [LIB] stat (utility.h:1165) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=25] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=116, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:25.268677] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF291-0-0] [lt=36] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=5488, avg_log_submit_to_slide_time=17049) [2024-04-22 16:23:25.268693] INFO [LIB] stat (utility.h:1165) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=32] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=10, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:25.271203] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10483][LSSysTblUp0][T0][YB42AC175881-000616AB0D2FECBC-0-0] [lt=62][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:25.285432] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=54] PNIO [ratelimit] time: 1713774205285428, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007600, add_bytes: 0 [2024-04-22 16:23:25.296676] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:47, local_retry_times:47, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.296764] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.296786] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.296798] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.296825] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.301825] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:25.301873] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.301923] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=47][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.301940] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.301955] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.301967] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.301978] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.302030] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.302103] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=62][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.302126] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.302152] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=48, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.302199] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30] will sleep(sleep_us=48000, remain_us=672725, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.313146] INFO [PALF] check_parent_health (log_config_mgr.cpp:2764) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35] re_register_parent reason(palf_id=1, self="172.23.88.129:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713774201308691, last_first_register_time_us=1713773712050886, register_time_us=1713774201308691, curr_time_us=1713774205313142) [2024-04-22 16:23:25.313300] INFO [PALF] check_parent_health (log_config_mgr.cpp:2769) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=89] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.129:2882") [2024-04-22 16:23:25.316744] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:173, local_retry_times:173, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.316853] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=72][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.316893] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=37][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.316908] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.316934] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.317504] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=59] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:25.318587] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318643] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=53] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318684] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318699] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318713] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318727] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318740] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318753] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318766] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318779] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318793] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318813] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318827] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318840] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318853] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.318865] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:25.320955] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=43][errcode=-4725] [2024-04-22 16:23:25.320998] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.321027] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.321043] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.321057] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.321103] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=43][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.321116] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.321164] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.321213] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.321229] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:25.321247] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=174, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.321281] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=9505861, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=174, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.350356] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:48, local_retry_times:48, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.350506] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=126][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.350529] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.350541] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.350566] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.355519] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:25.355565] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.355592] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.355614] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.355634] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.355652] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.355668] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.355729] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.355833] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=92][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.355865] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.355902] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=32][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=49, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.355954] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=33] will sleep(sleep_us=49000, remain_us=618969, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.405138] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=35][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.405228] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.405250] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.405261] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.405310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.409291] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:25.409347] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=54][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.409377] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.409394] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.409408] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.409419] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.409430] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.409516] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=47] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.409555] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.409568] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.409585] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=50, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.409620] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18] will sleep(sleep_us=50000, remain_us=565304, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.412652] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:25.412754] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774205256674059, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774205412639) [2024-04-22 16:23:25.412793] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774205212643, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:25.421483] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:174, local_retry_times:174, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.421612] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=100][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.421641] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.421660] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.421715] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.425104] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=52][errcode=-4725] [2024-04-22 16:23:25.425170] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=61][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.425213] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.425239] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.425255] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.425266] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.425277] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.425333] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.425385] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.425404] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:25.425430] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=175, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.425579] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=130] will sleep(sleep_us=100000, remain_us=9401564, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=175, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.459779] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=39][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:50, local_retry_times:50, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.459860] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.459881] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.459891] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.459916] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.462743] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C90-0-0] [lt=25][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=3009, log_task={header:{begin_lsn:{lsn:366781}, end_lsn:{lsn:366903}, log_id:3009, min_scn:{val:1713774205357661294, v:0}, max_scn:{val:1713774205357661294, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:366659}, prev_proposal_id:8, committed_end_lsn:{lsn:366781}, data_checksum:667309872, accum_checksum:2193849434, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774205362506, freeze_ts:1713774205362507, submit_ts:1713774205362519, flushed_ts:1713774205367394, gen_to_freeze cost time:1, gen_to_submit cost time:13, submit_to_flush cost time:4875}, fs_cb_begin_ts=1713774205462716, log_life_time=100210) [2024-04-22 16:23:25.463678] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:25.463729] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=48][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.463759] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.463781] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.463817] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=34][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.463829] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.463840] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.463900] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.463956] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.463977] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.463997] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=51, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.464031] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18] will sleep(sleep_us=51000, remain_us=510892, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.476654] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=60] Cache replace map node details(ret=0, replace_node_count=0, replace_time=10438, replace_start_pos=1195366, replace_num=62914) [2024-04-22 16:23:25.476709] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:25.485526] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=55] BlockGCTimerTask success(ret=0, cost_time_us=43, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.23.88.129:2882", log_dir:"/home/creality/oceanbase/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-04-22 16:23:25.515169] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:51, local_retry_times:51, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.515256] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.515285] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.515320] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=33][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.515362] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.518335] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=65] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:25.519426] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=32][errcode=-4725] [2024-04-22 16:23:25.519495] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=66][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.519523] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.519540] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.519554] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.519565] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.519576] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.519621] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.519660] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.519673] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.519708] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=33][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=52, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.519744] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19] will sleep(sleep_us=52000, remain_us=455180, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.525764] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:175, local_retry_times:175, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.525856] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.525878] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.525890] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.525914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.529825] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:25.529869] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.529900] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.529945] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=42][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.529966] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.529981] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.529992] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.530040] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.530077] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.530091] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:25.530107] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=176, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.530139] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=9297003, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=176, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.533990] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=63] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=15588, clean_start_pos=4655673, clean_num=125829) [2024-04-22 16:23:25.571897] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:52, local_retry_times:52, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.571983] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.572005] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.572016] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.572042] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.575486] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:25.575539] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=50][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.575580] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=37][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:25.575633] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=50][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.575655] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:25.575674] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:25.575692] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.575756] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.575828] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=63][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:25.575858] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=26][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.575885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=25][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=53, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.575934] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29] will sleep(sleep_us=53000, remain_us=398990, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.610884] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:249) [10820][T1_RFLWorker][T1][YB42AC175881-000616AB070FEBAD-0-0] [lt=50] ObRemoteFetchWorker is running(thread_index=0) [2024-04-22 16:23:25.612861] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:25.612955] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=88][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774205458539036, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774205612847) [2024-04-22 16:23:25.612993] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774205412822, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:25.613428] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2644) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] [PALF_DUMP](palf_id=1, self="172.23.88.129:2882", [SlidingWindow]={palf_id:1, self:"172.23.88.129:2882", lsn_allocator:{max_log_id:3010, max_lsn:367025, max_scn:{val:1713774205458539036, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:367025}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:358576}}, last_submit_lsn:{lsn:366903}, last_submit_end_lsn:{lsn:367025}, last_submit_log_id:3010, last_submit_log_pid:8, max_flushed_lsn:{lsn:366903}, max_flushed_end_lsn:{lsn:367025}, max_flushed_log_pid:8, committed_end_lsn:{lsn:367025}, last_slide_log_id:3010, last_slide_scn:{val:1713774205458539036, v:0}, last_slide_lsn:{lsn:366903}, last_slide_end_lsn:{lsn:367025}, last_slide_log_pid:8, last_slide_log_accum_checksum:4239111920, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f3cd2de4530}, [StateMgr]={this:0x7f3cd2de8330, self:"172.23.88.129:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, leader:"172.23.88.130:2882", leader_epoch:11, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:358576}, scan_disk_log_finished:true, last_check_start_id:3011, is_changing_config_with_arb:false, reconfirm_start_time_us:1713774181765332, palf_role_change_cb:0x7f3cd2deebf0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.23.88.129:2882", alive_paxos_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_end_lsn:{lsn:356581}, prev_mode_pid:7}, reconfig_barrier:{prev_log_proposal_id:6, prev_lsn:{lsn:356459}, prev_end_lsn:{lsn:356581}, prev_mode_pid:7}, persistent_config_version:{proposal_id:8, config_seq:9}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1713774205313238, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1713774205313238, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f3cd2de5c70}, [ModeMgr]={palf_id:1, self:"172.23.88.129:2882", applied_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:8}, log_config_meta:{version:3, proposal_id:8, prev:{version:1, config:{config_version:{proposal_id:7, config_seq:8}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:8, config_seq:9}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:7, prev_lsn:{lsn:358454}, prev_mode_pid:8}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:8, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:114688}, readable_log_tail:{lsn:114688}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:234, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66990080, block_header_serialize_buf_:0x7f3cd2dea4b8, flashback_version:0}, log_storage:{log_tail:{lsn:367025}, readable_log_tail:{lsn:367025}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/log", dir_fd:235, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66737743, block_header_serialize_buf_:0x7f3cd2dec138, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1713774181764894, this:0x7f3cd2de8a70}, [Reconfirm]={palf_id:1, self:"172.23.88.129:2882", state:"FINISHED", new_proposal_id:7, prepare_log_ack_list:1"172.23.88.131:2882";, curr_paxos_follower_list:2{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.23.88.129:2882", majority_max_accept_pid:6, majority_max_lsn:{lsn:356581}, saved_end_lsn:{lsn:356581}, last_submit_prepare_req_time_us:1713774181765723, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f3cd2de84b0}) [2024-04-22 16:23:25.614017] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=578] [PALF_DUMP](palf_id=1, self="172.23.88.129:2882", ack_info_list=[{member:{server:"172.23.88.129:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.130:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.131:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}]) [2024-04-22 16:23:25.614067] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4287) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=45] current log_task status(palf_id=1, self="172.23.88.129:2882", log_id=3011, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"172.23.88.129:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.130:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}, {member:{server:"172.23.88.131:2882", timestamp:1, flag:0}, last_ack_time_us:1713774192197896, last_flushed_end_lsn:{lsn:358576}}]) [2024-04-22 16:23:25.629180] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:53, local_retry_times:53, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:25.629303] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=95][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.629385] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=78][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:25.629405] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:25.629538] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=91][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:25.630324] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:25.632387] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=83][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:25.632533] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.632601] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=54000, remain_us=342323, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.633601] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.633726] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=9193417, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=177, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.662773] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C92-0-0] [lt=91] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:25.662814] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C92-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1098, avg_group_log_size=122) [2024-04-22 16:23:25.662831] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C92-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=1098) [2024-04-22 16:23:25.676888] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:25.690645] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.690703] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFDEE-0-0] [lt=69][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774205690612}) [2024-04-22 16:23:25.690759] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=55000, remain_us=284164, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.710465] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:310) [10850][T1_ArcFetcher][T1][YB42AC175881-000616AB076FEBAD-0-0] [lt=44] ObArchiveFetcher is running(thread_index=0) [2024-04-22 16:23:25.712987] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:25.734235] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35CA-0-0] [lt=261][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:25.734305] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35CA-0-0] [lt=66][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:25.734363] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35CA-0-0] [lt=53][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:25.734403] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35CA-0-0] [lt=39][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:25.734652] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20723859456, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:25.734811] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=49] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:25.735722] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [10462][pnio1][T0][YB42AC175881-000616AB0B5FEDE1-0-0] [lt=31] ts response task statistics(alloc_count=3702, free_count=1850) [2024-04-22 16:23:25.737958] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.738049] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=9089094, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=178, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.749641] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.749788] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=56000, remain_us=225136, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.762829] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C94-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=3011, log_task={header:{begin_lsn:{lsn:367025}, end_lsn:{lsn:367147}, log_id:3011, min_scn:{val:1713774205647675348, v:0}, max_scn:{val:1713774205647675348, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:366903}, prev_proposal_id:8, committed_end_lsn:{lsn:367025}, data_checksum:667309872, accum_checksum:423008254, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774205662765, freeze_ts:1713774205662765, submit_ts:1713774205662801, flushed_ts:1713774205667092, gen_to_freeze cost time:0, gen_to_submit cost time:36, submit_to_flush cost time:4291}, fs_cb_begin_ts=1713774205762815, log_life_time=100050) [2024-04-22 16:23:25.809808] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.809903] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=1] will sleep(sleep_us=57000, remain_us=165021, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.813096] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=132][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:25.813144] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:25.813179] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774205458539036, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774205813080) [2024-04-22 16:23:25.813202] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774205613017, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:25.817513] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=89] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:25.842273] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.842369] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=8984773, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=179, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.870620] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.870667] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0] will sleep(sleep_us=58000, remain_us=104256, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713774205974922) [2024-04-22 16:23:25.877077] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=66] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:25.932786] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=0][errcode=-4012] [2024-04-22 16:23:25.932830] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=41][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:25.932863] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=30][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:23:25.932887] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=21][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:25.932912] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=22][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:23:25.932996] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=81][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:23:25.933008] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=11][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:25.933020] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:25.933032] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=9][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:25.933052] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:25.933084] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=29][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:23:25.933153] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=55][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:25.933166] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:25.933177] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=10][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:23:25.933233] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=43][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:23:25.933274] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=35] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1956792) [2024-04-22 16:23:25.933298] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=23][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:23:25.933318] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=16][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:25.933333] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:23:25.933354] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:25.933376] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:25.933427] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=44][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:25.933592] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=78][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:23:25.933626] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=32][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:25.933648] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=19][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:23:25.933663] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=12][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:25.933683] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=18][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:25.933698] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:521) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-04-22 16:23:25.933714] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:236) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBEF-0-0] [lt=14][errcode=-4012] fail to process refresh task(ret=-4012, ret="OB_TIMEOUT", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-04-22 16:23:25.933767] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:599) [10477][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4012] fail to batch process task(ret=-4012) [2024-04-22 16:23:25.933779] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:499) [10477][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] fail to batch execute task(ret=-4012, tasks.count()=1) [2024-04-22 16:23:25.933916] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:261) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=71] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:25.933966] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:305) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=44] [REFRESH_SCHEMA] begin refresh schema, (begin_time=1713774205933965) [2024-04-22 16:23:25.933981] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2388) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[]) [2024-04-22 16:23:25.935153] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=52] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:25.938760] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.938848] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] already timeout, do not need sleep(sleep_us=0, remain_us=1995161, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774207934009) [2024-04-22 16:23:25.942582] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.942676] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=1000, remain_us=1991335, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:25.946218] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.946301] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8880841, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=180, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:25.946285] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.946370] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=2000, remain_us=1987641, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:25.952766] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.952850] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=3000, remain_us=1981161, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:25.959035] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.959102] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=4000, remain_us=1974908, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:25.963229] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C96-0-0] [lt=44][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=3012, log_task={header:{begin_lsn:{lsn:367147}, end_lsn:{lsn:367269}, log_id:3012, min_scn:{val:1713774205755269741, v:0}, max_scn:{val:1713774205755269741, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:367025}, prev_proposal_id:8, committed_end_lsn:{lsn:367147}, data_checksum:667309872, accum_checksum:3338165330, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774205762790, freeze_ts:1713774205762790, submit_ts:1713774205762797, flushed_ts:1713774205768973, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:6176}, fs_cb_begin_ts=1713774205963180, log_life_time=200390) [2024-04-22 16:23:25.967212] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.967275] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=5000, remain_us=1966735, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:25.968798] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=29] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=4965, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:25.968859] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=976) [2024-04-22 16:23:25.968910] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=86, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:25.969048] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=58] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=87, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:25.976197] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.976280] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=6000, remain_us=1957731, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:25.985812] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.985919] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=7000, remain_us=1948092, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:25.996715] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:25.996801] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=8000, remain_us=1937210, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.008755] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.008841] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=9000, remain_us=1925170, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.013317] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:26.013502] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=78][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774205647675348, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774206013299) [2024-04-22 16:23:26.013555] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774205813220, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:26.021494] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.021578] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=10000, remain_us=1912432, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.034987] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.035076] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=11000, remain_us=1898934, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.050242] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.050249] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.050343] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=12000, remain_us=1883667, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.050366] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8776777, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=181, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.066271] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.066356] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=13000, remain_us=1867655, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.077282] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:26.083096] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.083182] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1850829, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.101177] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.101264] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=1832747, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.111414] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=73] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:26.119363] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.119505] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=16000, remain_us=1814505, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.135920] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=72] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:26.138245] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.138330] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=17000, remain_us=1795681, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.153717] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.153834] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8673308, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=182, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.158484] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.158584] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=18000, remain_us=1775427, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.180601] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.180705] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=1753305, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.190369] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.190531] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.190576] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=77] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.191200] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.191217] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=67] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.191268] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.191319] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.191492] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.191703] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=44] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.192017] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.192488] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.192849] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=58] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.192901] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=52] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.193133] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.193227] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=48] gc stale ls task succ [2024-04-22 16:23:26.193775] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.194109] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.194584] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=27] ====== check clog disk timer task ====== [2024-04-22 16:23:26.194635] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=39] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:26.194681] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=20] cannot_recycle_log_size statistics(cannot_recycle_log_size=367513, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:26.194762] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=46] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:26.194886] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=33] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:26.194950] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=63] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.194990] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=56] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=11013, mem_used=79030272) [2024-04-22 16:23:26.195265] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.195504] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=34] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.196410] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.196820] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=65] current monitor number(seq_=-1) [2024-04-22 16:23:26.197482] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.197534] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.197539] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=61] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.199199] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:26.202702] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:26.202763] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:26.202785] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:26.202805] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:26.203726] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.203813] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=20000, remain_us=1730198, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.213632] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:26.213710] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=73][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774205955458522, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774206213616) [2024-04-22 16:23:26.213770] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774206013582, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:26.218843] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:26.218934] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=57] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:23:26.218962] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=27] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:23:26.228014] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.228090] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=21000, remain_us=1705920, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.233846] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=57] PNIO [ratelimit] time: 1713774206233843, bytes: 66503965, bw: 0.030454 MB/s, add_ts: 1004577, add_bytes: 32080 [2024-04-22 16:23:26.234177] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=75][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:490, tid:10477}]) [2024-04-22 16:23:26.235060] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=9][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:26.235141] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=57] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:26.235185] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=38][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:26.235213] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=25][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:26.235243] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=28][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:26.235262] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:26.235275] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:26.235292] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:26.235329] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=34][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:26.235341] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:26.235352] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=9][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:26.235363] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:26.235375] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:26.235387] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:26.235398] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:26.235423] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:26.235472] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:26.235491] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=16][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:26.235503] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:26.235515] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:26.235540] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=21][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:26.235553] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.235586] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=27][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:26.235648] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=52][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:26.235681] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=28][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:26.235701] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=18][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:26.235733] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8D-0-0] [lt=17][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:26.235784] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=42][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:26.235812] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=24][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:26.235834] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=19][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:26.235849] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=14][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:26.235876] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=25][errcode=-5019] query failed(ret=-5019, conn=0x7f3cc12b0050, start=1713774206234787, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:26.235896] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=18][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:26.235909] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=10][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:26.236005] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=13][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:26.236035] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8D-0-0] [lt=28][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:26.239788] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35CD-0-0] [lt=222][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:26.239853] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35CD-0-0] [lt=61][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:26.239901] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35CD-0-0] [lt=45][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:26.239936] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35CD-0-0] [lt=34][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:26.249245] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=42][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:21, local_retry_times:21, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.249295] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=37][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.249307] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.249314] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=6][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.249341] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.252763] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] [2024-04-22 16:23:26.252791] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=27][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.252839] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=46][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.252858] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.252876] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.252891] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.252899] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.252940] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.252963] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.252971] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=7][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.252981] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=9][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=22, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.253002] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11] will sleep(sleep_us=22000, remain_us=1681008, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.253998] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:182, local_retry_times:182, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.254107] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=81][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.254165] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.254186] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.254213] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.257278] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=13] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:26.258264] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:26.258311] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.258348] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.258373] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.258393] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.258410] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.258425] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.258516] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=50] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.258570] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.258584] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:26.258601] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=183, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.258635] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=8568508, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=183, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.259489] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=200][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:26.259578] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=68] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:26.259636] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=53][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:26.259665] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=26][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:26.259693] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=25][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:26.259719] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=19][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:26.259735] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:26.259752] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:26.259764] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:26.259775] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:26.259786] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:26.259800] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=12][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:26.259818] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=16][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:26.259852] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=31][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:26.259870] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=16][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:26.259902] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=19][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:26.259921] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=16][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:26.259942] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:26.259954] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:26.259966] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:26.259980] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:26.259993] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.260017] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:26.260042] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:26.260052] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:26.260074] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=21][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:26.260106] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:26.260122] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:26.260138] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:26.260151] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:26.260163] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:26.260176] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1498050, start=1713774206259226, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:26.260194] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:26.260207] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:26.260296] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=14][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:26.260333] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=34][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:26.260367] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=32][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:26.260383] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=15][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:26.260398] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=11][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:26.260413] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=12][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:26.260425] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED89-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:26.275139] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:22, local_retry_times:22, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.275220] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.275248] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.275261] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.275285] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.276822] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0D1FED59-0-0] [lt=62][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:26.277463] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:23:26.279172] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=95][errcode=-4725] [2024-04-22 16:23:26.279213] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.279252] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=35][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.279276] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.279296] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.279313] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.279330] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.279393] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.279499] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=96][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.279524] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=22][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.279551] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=23, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.279584] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19] will sleep(sleep_us=23000, remain_us=1654427, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.293149] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=96] PNIO [ratelimit] time: 1713774206293146, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007718, add_bytes: 0 [2024-04-22 16:23:26.302748] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:23, local_retry_times:23, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.302836] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.302867] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.302885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.302928] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.307983] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=38][errcode=-4725] [2024-04-22 16:23:26.308033] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=48][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.308065] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.308085] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.308105] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.308146] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=39][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.308165] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.308227] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.308321] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=85][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.308354] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=29][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.308381] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=24, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.308418] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=22] will sleep(sleep_us=24000, remain_us=1625593, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.320849] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.320913] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.320940] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.320956] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.320970] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321005] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321019] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321032] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321045] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321058] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321071] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321084] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321097] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321110] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321130] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.321144] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:26.332587] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=50][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:24, local_retry_times:24, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.332660] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=54][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.332686] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.332704] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.332753] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=35][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.336714] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:26.338005] INFO [COMMON] wash (ob_kvcache_store.cpp:356) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Wash time detail, (compute_wash_size_time=298, refresh_score_time=129, wash_time=0) [2024-04-22 16:23:26.338810] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:26.338858] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.338900] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=38][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.338925] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.338946] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.338963] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.338981] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.339045] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.339097] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=43][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.339116] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.339141] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=25, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.339209] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=50] will sleep(sleep_us=25000, remain_us=1594802, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.358760] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:183, local_retry_times:183, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.358848] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.358870] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.358882] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.358906] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.361527] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [10717][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=89] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):64, sequence:2}) [2024-04-22 16:23:26.364220] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:26.364263] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.364293] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.364311] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.364347] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.364359] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.364371] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.364374] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:25, local_retry_times:25, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.364434] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=42][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.364421] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.364490] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=53][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.364501] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=72][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.364502] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.364515] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:26.364541] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=184, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.364547] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.364583] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25] will sleep(sleep_us=100000, remain_us=8462559, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=184, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.369645] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF298-0-0] [lt=158] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=28, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:26.369695] INFO [PALF] read (log_cache.cpp:100) [10866][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=854, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:26.369713] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF298-0-0] [lt=65] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=61047, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:26.369797] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:26.369782] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF298-0-0] [lt=65] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=15, avg_log_submit_to_flush_time=5195, avg_log_submit_to_slide_time=61032) [2024-04-22 16:23:26.369842] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=43][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.369866] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=21][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.369881] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.369895] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.369906] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.369916] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.369916] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=166] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=98, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:26.369956] INFO [LIB] stat (utility.h:1165) [10816][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=39] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=10, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:26.369967] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.370003] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=29][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.370017] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.370034] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=26, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.370067] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18] will sleep(sleep_us=26000, remain_us=1563943, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.396323] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=32][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:26, local_retry_times:26, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.396411] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=68][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.396488] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=73][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.396503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.396528] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.400423] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:26.400505] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=80][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.400536] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.400553] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.400572] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.400590] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.400647] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=54][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.400710] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.400762] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=44][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.400784] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.400804] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=27, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.400836] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18] will sleep(sleep_us=27000, remain_us=1533174, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.414015] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=177][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:26.414087] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=68][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774206154837760, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774206414002) [2024-04-22 16:23:26.414124] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774206213951, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:26.428108] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:27, local_retry_times:27, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.428190] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.428218] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.428237] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.428270] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.432176] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=43][errcode=-4725] [2024-04-22 16:23:26.432231] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=52][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.432273] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=38][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.432293] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.432308] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.432319] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.432330] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.432377] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.432415] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.432428] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.432489] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=58][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=28, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.432543] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=37] will sleep(sleep_us=28000, remain_us=1501467, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.460720] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:28, local_retry_times:28, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.460803] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.460832] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.460845] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.460892] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.464666] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:26.464711] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=42][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.464743] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.464762] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.464775] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.464787] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.464764] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:184, local_retry_times:184, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.464819] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=30][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.464855] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.464890] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.464867] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.464904] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.464918] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.464928] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.464938] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.464963] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=29, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.465007] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=27] will sleep(sleep_us=29000, remain_us=1469003, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.468702] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:26.468745] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.468784] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=34][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.468807] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.468826] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.468843] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.468858] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.468921] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.469064] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=133][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.469087] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:26.469113] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=185, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.469229] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=99] will sleep(sleep_us=100000, remain_us=8357914, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=185, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.477641] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:23:26.494192] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=41][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:29, local_retry_times:29, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.494297] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=79][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.494327] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.494342] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.494368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.498278] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=48][errcode=-4725] [2024-04-22 16:23:26.498315] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.498343] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.498365] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.498381] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.498392] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.498403] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.498490] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.498564] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=64][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.498587] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.498613] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=30, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.498668] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=38] will sleep(sleep_us=30000, remain_us=1435342, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.528911] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:30, local_retry_times:30, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.529003] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.529040] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.529061] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.529124] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.532934] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=28][errcode=-4725] [2024-04-22 16:23:26.532977] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:26.533009] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:26.533032] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:26.533054] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:26.533088] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=32][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:26.533110] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=21][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.533166] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.533239] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=63][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:26.533269] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=26][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:26.533310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=37][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=31, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.533358] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=33] will sleep(sleep_us=31000, remain_us=1400652, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.538790] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=107] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:26.564508] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:31, local_retry_times:31, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:26.564591] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.564619] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:26.564632] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:26.564681] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=34][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:26.568568] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.568696] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=32000, remain_us=1365314, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.573371] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.573490] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=8253652, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=186, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.603604] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.603698] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=33000, remain_us=1330313, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.614246] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:26.614347] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774206359050217, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774206614232) [2024-04-22 16:23:26.614379] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774206414171, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:26.636942] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=27][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:26.639211] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.639299] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=34000, remain_us=1294712, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.663892] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C9E-0-0] [lt=64] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:26.663986] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C9E-0-0] [lt=63] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=6, avg_log_batch_cnt=0, total_group_log_size=732, avg_group_log_size=122) [2024-04-22 16:23:26.664052] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C9E-0-0] [lt=60] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=732) [2024-04-22 16:23:26.664134] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3C9E-0-0] [lt=37][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=3016, log_task={header:{begin_lsn:{lsn:367635}, end_lsn:{lsn:367757}, log_id:3016, min_scn:{val:1713774206551487259, v:0}, max_scn:{val:1713774206551487259, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:367513}, prev_proposal_id:8, committed_end_lsn:{lsn:367635}, data_checksum:667309872, accum_checksum:124563987, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774206563795, freeze_ts:1713774206563796, submit_ts:1713774206563808, flushed_ts:1713774206570322, gen_to_freeze cost time:1, gen_to_submit cost time:13, submit_to_flush cost time:6514}, fs_cb_begin_ts=1713774206664107, log_life_time=100312) [2024-04-22 16:23:26.673643] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:26.677055] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.677144] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=35000, remain_us=1256867, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.677278] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.677365] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8149777, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=187, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.677861] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=67] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:23:26.714337] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:26.716197] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.716260] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=36000, remain_us=1217750, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.716272] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFE20-0-0] [lt=56][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774206716174}) [2024-04-22 16:23:26.735356] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D0-0-0] [lt=211][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:26.735424] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D0-0-0] [lt=64][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:26.735514] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D0-0-0] [lt=85][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:26.735618] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D0-0-0] [lt=101][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:26.737839] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=52] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:26.739514] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=85] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:26.756873] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.756959] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=37000, remain_us=1177051, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.764146] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3CA0-0-0] [lt=82][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=3017, log_task={header:{begin_lsn:{lsn:367757}, end_lsn:{lsn:367879}, log_id:3017, min_scn:{val:1713774206657251605, v:0}, max_scn:{val:1713774206657251605, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:367635}, prev_proposal_id:8, committed_end_lsn:{lsn:367757}, data_checksum:667309872, accum_checksum:2432380170, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774206663879, freeze_ts:1713774206663879, submit_ts:1713774206663960, flushed_ts:1713774206668033, gen_to_freeze cost time:0, gen_to_submit cost time:81, submit_to_flush cost time:4073}, fs_cb_begin_ts=1713774206764120, log_life_time=100241) [2024-04-22 16:23:26.781135] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.781225] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=8045917, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=188, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.797259] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.797312] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=38000, remain_us=1136698, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.814480] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=47][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:26.814558] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=73][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:26.814597] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774206551487259, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774206814429) [2024-04-22 16:23:26.814622] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774206614401, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:26.839258] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.839338] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=39000, remain_us=1094672, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.878040] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:23:26.883085] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.883179] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=1050832, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.885593] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.885674] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=7941468, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=189, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:26.913164] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=121][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:26.913205] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=29] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:26.913223] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=15][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:26.913244] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=20][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:26.913253] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=8][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:26.913261] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:26.913267] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:26.913274] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:23:26.913280] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:26.913285] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:26.913289] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:26.913294] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:26.913300] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:26.913306] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:26.913311] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:26.913323] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=6][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:26.913334] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:26.913341] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:26.913346] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:26.913352] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882, ret=-5019) [2024-04-22 16:23:26.913359] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:26.913364] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:26.913377] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=10][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:26.913390] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:26.913395] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:26.913399] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:26.913414] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:26.913423] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10708][T1_Occam][T1][YB42AC175881-000616AB087FEC15-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-04-22 16:23:26.913430] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:26.913455] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:26.913461] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:26.913467] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f3cbe430050, start=1713774206913045, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:26.913474] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:26.913480] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f3ce8856298, table=__all_server, condition=where svr_ip='172.23.88.129' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.129' and svr_port=2882, columns_str="zone") [2024-04-22 16:23:26.913493] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='172.23.88.129' and svr_port=2882) [2024-04-22 16:23:26.913549] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:635) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f3ce8856298, where_condition="where svr_ip='172.23.88.129' and svr_port=2882", zone_name_holder=) [2024-04-22 16:23:26.913557] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:568) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:23:26.913564] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:577) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:23:26.913577] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:26.927546] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.927789] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=41000, remain_us=1006221, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.940288] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=72] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:26.964381] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3CA2-0-0] [lt=88][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=3018, log_task={header:{begin_lsn:{lsn:367879}, end_lsn:{lsn:368001}, log_id:3018, min_scn:{val:1713774206759927235, v:0}, max_scn:{val:1713774206759927235, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:367757}, prev_proposal_id:8, committed_end_lsn:{lsn:367879}, data_checksum:667309872, accum_checksum:919348549, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774206764073, freeze_ts:1713774206764073, submit_ts:1713774206764085, flushed_ts:1713774206768236, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:4151}, fs_cb_begin_ts=1713774206964354, log_life_time=200281) [2024-04-22 16:23:26.969199] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=4787, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:26.969254] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=732) [2024-04-22 16:23:26.969303] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=79, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:26.969429] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=69] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=76, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:26.972268] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.972356] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=42000, remain_us=961654, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:26.989711] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:26.989798] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7837345, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=190, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.014655] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:27.014776] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=115][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774206657251605, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774207014641) [2024-04-22 16:23:27.014833] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774206814642, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:27.018484] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.018572] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=43000, remain_us=915438, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.065548] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.065666] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=44000, remain_us=868345, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.078204] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=79] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:23:27.093498] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.093586] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=7733557, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=191, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.112580] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:27.113565] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.113650] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=45000, remain_us=820360, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.140933] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=67] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:27.162913] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.163007] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=46000, remain_us=771003, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.190344] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.190559] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.190583] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=65] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.190599] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.190694] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.190858] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.190969] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.190978] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:469) [10838][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=49] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-04-22 16:23:27.191335] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=78] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.191486] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.192099] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC13-0-0] [lt=12][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-04-22 16:23:27.192139] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC13-0-0] [lt=38][errcode=-4002] refresh sever list failed(ret=-4002) [2024-04-22 16:23:27.192152] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.192156] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [10407][ServerTracerTim][T0][YB42AC175881-000616AB07CFEC13-0-0] [lt=15][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-04-22 16:23:27.192168] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.192807] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.193122] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.193338] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=25] gc stale ls task succ [2024-04-22 16:23:27.193516] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=60] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.193997] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=68] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.194055] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.194226] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.194528] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=27] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-22 16:23:27.194530] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=86] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.194617] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [10847][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=63] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=101) [2024-04-22 16:23:27.194641] INFO pn_ratelimit (group.c:589) [10608][IngressService][T0][Y0-0000000000000000-0-0] [lt=26] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:23:27.194766] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=16] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-22 16:23:27.194829] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=48] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-22 16:23:27.194855] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [10846][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=22] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=101) [2024-04-22 16:23:27.194930] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=55] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:27.195115] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=55] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:27.195161] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=11053, mem_used=79030272) [2024-04-22 16:23:27.195281] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:306) [10844][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=31] ====== traversal_flush timer task ====== [2024-04-22 16:23:27.195540] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.195489] INFO [MDS] flush (mds_table_mgr.cpp:172) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC11-0-0] [lt=49] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774217210958129, latest_log_ts:{val:1713774200564621708, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:27.195658] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:338) [10844][T1_Flush][T1][YB42AC175881-000616AB088FEC11-0-0] [lt=159] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-22 16:23:27.195881] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=77] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.197220] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=43] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.197475] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.197516] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7629626, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=192, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.197525] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.197549] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10519][AutoLSLoc][T0][YB42AC175881-000616AB0ACFEC0E-0-0] [lt=179] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:27.198398] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.198526] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:27.199569] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=12][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:27.199644] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:27.199675] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:23:27.199695] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:27.199714] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:23:27.199733] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:27.199746] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:27.199763] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:23:27.199776] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:27.199787] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:27.199799] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:27.199831] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=30][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:23:27.199842] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=10][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:23:27.199853] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=10][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:23:27.199864] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:27.199876] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:27.199888] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:27.199900] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:27.199925] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:27.199938] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:27.199952] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:27.199964] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:27.199975] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:23:27.200000] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=22][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:27.200012] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.200038] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:27.200064] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:27.200075] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:27.200085] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:27.200107] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10899][BlackListServic][T1][YB42AC175881-000616AB07BFEC54-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:27.200125] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10899][BlackListServic][T0][YB42AC175881-000616AB07BFEC54-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:23:27.200140] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:27.200169] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:27.200182] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:27.200196] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cf3bc0050, start=1713774207199140, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:27.200215] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:27.200227] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:23:27.200321] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [10899][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14] ls blacklist refresh finish(cost_time=3250) [2024-04-22 16:23:27.201391] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-22 16:23:27.202993] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:131) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] ====== checkpoint timer task ====== [2024-04-22 16:23:27.203203] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:27.203257] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:27.203276] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:27.203324] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=44][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:27.203391] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=94][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:27.203522] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=114] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-04-22 16:23:27.203576] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=48][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-04-22 16:23:27.203616] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=36][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-04-22 16:23:27.203648] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=29][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-04-22 16:23:27.203674] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=19][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:27.203692] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=17][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:27.203715] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=15][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-04-22 16:23:27.203734] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=17][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:27.203748] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:27.203787] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=36][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:27.203800] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=12][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:27.203812] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:27.203824] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:27.203835] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:27.203858] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:27.203871] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:27.203887] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:27.203935] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:27.203972] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=34][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:27.203996] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=20][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:27.204010] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=12][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.204050] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=35][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:27.204076] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:27.204088] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:27.204097] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:27.204120] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:27.204136] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10801][T1_ReqMemEvict][T1][YB42AC175881-000616AB086FEC12-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2024-04-22 16:23:27.204151] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:23:27.204165] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:27.204177] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:27.204191] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3d1177a050, start=1713774207203136, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:23:27.204225] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:27.204241] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:479) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:23:27.204336] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:141) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2024-04-22 16:23:27.204368] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:252) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-5019] failed to fetch max rule version(ret=-5019) [2024-04-22 16:23:27.204388] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:95) [10801][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2024-04-22 16:23:27.204699] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1426) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=26] succ to get rec scn(*this={this:0x7f3cd1c04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681662748, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:27.204744] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=43] tx ctx memtable get rec scn(this={ObITable:{this:0x7f3cd10be080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f3cd10be080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:23:27.204779] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:856) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-22 16:23:27.204881] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=41] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f3cd2d60510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774217210958129, latest_log_ts:{val:1713774200564621708, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:23:27.204970] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:599) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=85] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1}) [2024-04-22 16:23:27.204995] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:470) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2024-04-22 16:23:27.205013] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] get rec log scn(service_type_=0, rec_log_ts={val:1713774200564621708, v:0}) [2024-04-22 16:23:27.205026] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-22 16:23:27.205037] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=2, rec_log_ts={val:1713773717210958135, v:0}) [2024-04-22 16:23:27.205057] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:164) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713773717210958135, v:0}, checkpoint_scn_in_ls_meta={val:1713773717210958135, v:0}, ls_id={id:1}, service_type="DAS_ID") [2024-04-22 16:23:27.205091] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1579) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-22 16:23:27.205112] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:185) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-22 16:23:27.205124] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:192) [10843][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-22 16:23:27.205353] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC47-0-0] [lt=15][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:27.205418] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC47-0-0] [lt=49] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:23:27.205483] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC47-0-0] [lt=60][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:23:27.205502] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC47-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:23:27.205521] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC47-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:23:27.205538] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC47-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:27.205551] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC47-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:27.205567] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10404][ServerGTimer][T1][YB42AC175881-000616AB07DFEC47-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:27.209094] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC7A-0-0] [lt=29][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:27.209167] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC7A-0-0] [lt=55] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:27.209226] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC7A-0-0] [lt=54][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:27.210865] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1918) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=116] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-22 16:23:27.210939] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1459) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=67] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:27.210975] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1476) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=32] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=214748360, cache_obj_num=6, cache_node_num=4) [2024-04-22 16:23:27.211003] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2682) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=24] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:27.211649] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC7B-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:27.211752] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC7B-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:27.213067] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.213150] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=47000, remain_us=720860, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.213658] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [10740][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=109] ====== tenant freeze timer task ====== [2024-04-22 16:23:27.214403] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [10826][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-04-22 16:23:27.214812] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713774207058711777, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=156096, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1713774207214807, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-22 16:23:27.214916] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:27.215003] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774207058711777, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774207214901) [2024-04-22 16:23:27.215062] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774207014855, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:27.220367] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2401) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=17] Dumping All Cache Objs(alloc_obj_list.count()=6, alloc_obj_list=[{obj_id:99, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175160}, {obj_id:100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174200}, {obj_id:101, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173640}, {obj_id:102, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173992}, {obj_id:103, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:176531}, {obj_id:104, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:175513}]) [2024-04-22 16:23:27.220425] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2690) [10738][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=54] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:23:27.236318] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=50] PNIO [ratelimit] time: 1713774207236314, bytes: 66533860, bw: 0.028440 MB/s, add_ts: 1002471, add_bytes: 29895 [2024-04-22 16:23:27.237050] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=58][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:92, tid:10410}]) [2024-04-22 16:23:27.238291] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=17][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:27.238477] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=104] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:27.238511] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:27.238531] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:27.238551] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:27.238569] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:27.238589] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=18][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:27.238642] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=43][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:27.238661] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=16][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:27.238720] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=57][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:27.238731] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:27.238742] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:27.238754] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:27.238766] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:27.238777] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:27.238801] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:27.238815] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:27.238829] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:27.238845] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=14][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:27.238879] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=30][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:27.238900] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=18][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:27.238919] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=17][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.238956] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=30][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:27.238994] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=31][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:27.239012] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:27.239028] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=16][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:27.239072] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8E-0-0] [lt=15][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:27.239110] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=32][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:27.239151] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=36][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:27.239191] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=37][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:27.239205] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=13][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:27.239219] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd1460050, start=1713774207238015, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:27.239238] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=18][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:27.239252] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=10][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:27.239254] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=51] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=150, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-22 16:23:27.239301] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=44] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713774206737761], gts:1713774206735588933, latest_srr:[mts=1713774207239167]}) [2024-04-22 16:23:27.239350] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=13][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:27.239362] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8E-0-0] [lt=11][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:27.241128] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D3-0-0] [lt=188][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:27.241186] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D3-0-0] [lt=54][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:27.241236] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D3-0-0] [lt=46][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:27.241272] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D3-0-0] [lt=35][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:27.261853] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=49] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.130:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:8, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:2, zone:"zone2", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194674519, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713774194673374, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.130:2882", "172.23.88.131:2882", "172.23.88.129:2882"]) [2024-04-22 16:23:27.262927] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=110][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:27.262967] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=29] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:23:27.262987] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=17][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:23:27.263012] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=24][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:23:27.263026] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:23:27.263037] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:27.263044] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:27.263052] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:23:27.263059] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:27.263064] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:27.263070] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:27.263076] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:27.263082] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:27.263088] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:27.263094] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:27.263112] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:27.263119] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:27.263127] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=6][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:27.263133] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:27.263138] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:23:27.263146] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:27.263152] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.263167] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=11][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:27.263180] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:27.263186] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:27.263191] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:27.263207] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:27.263214] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=7][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:23:27.263221] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:27.263228] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=6][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:27.263234] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:27.263241] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=6][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd227a050, start=1713774207262795, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:27.263249] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=8][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:27.263256] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:23:27.263308] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=7][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:23:27.263316] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=7][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:23:27.263324] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=7][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:27.263335] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=10][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:27.263342] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:27.263350] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=6][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:23:27.263356] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [10829][T1_LSMetaCh][T1][YB42AC175881-000616AB07EFED8A-0-0] [lt=5][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:23:27.264333] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.264490] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=48000, remain_us=669522, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.264583] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3CA8-0-0] [lt=164][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.129:2882", log_id=3021, log_task={header:{begin_lsn:{lsn:368245}, end_lsn:{lsn:368367}, log_id:3021, min_scn:{val:1713774207160339269, v:0}, max_scn:{val:1713774207160339269, v:0}, data_len:66, proposal_id:8, prev_lsn:{lsn:368123}, prev_proposal_id:8, committed_end_lsn:{lsn:368245}, data_checksum:667309872, accum_checksum:2414125288, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713774207164394, freeze_ts:1713774207164395, submit_ts:1713774207164412, flushed_ts:1713774207168078, gen_to_freeze cost time:1, gen_to_submit cost time:18, submit_to_flush cost time:3666}, fs_cb_begin_ts=1713774207264554, log_life_time=100160) [2024-04-22 16:23:27.278352] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:23:27.281900] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10484][LSMetaTblUp0][T0][YB42AC175881-000616AB0D1FEDBC-0-0] [lt=63][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:27.300656] INFO eloop_run (eloop.c:140) [10465][pnio2][T0][Y0-0000000000000000-0-0] [lt=56] PNIO [ratelimit] time: 1713774207300653, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007507, add_bytes: 0 [2024-04-22 16:23:27.303528] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.303618] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=7523525, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=193, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.311820] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=104] tx gc loop thread is running(MTL_ID()=1) [2024-04-22 16:23:27.311877] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=54] try gc retain ctx [2024-04-22 16:23:27.311933] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:748) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] get_max_applied_scn(scn={val:1713774187421454783, v:0}, this={ls_id_:{id:1}, role_:2, proposal_id_:7, palf_committed_end_lsn_:{lsn:358576}, palf_committed_end_scn_:{val:1713774187421454783, v:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1713774187421454783, v:0}}) [2024-04-22 16:23:27.312053] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1023) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=99] get_min_unreplayed_log_info(lsn={lsn:368489}, scn={val:1713774207260564936, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713774207277752, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:368489}, next_to_submit_scn_:{val:1713774207260564936, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f3cd2ce7000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:368367}, end_lsn:{lsn:368489}, read_buf:{buf_len_:8192, buf_:0x7f3cd2ce7000}, block_size:67104768, log_storage_:0x7f3cd2deb4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713774207260564935, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713774207260564935, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:895831536, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713774207260564935, v:0}}}}}) [2024-04-22 16:23:27.314944] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1713774207260564935, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1713774207314889}, version_delta=54378) [2024-04-22 16:23:27.315238] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:253) [10859][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=221] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713774217210958129, latest_log_ts:{val:1713774200564621708, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:8, palf_env_:0x7f3ce8ff4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:23:27.315375] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [10719][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=77] LogLoopThread round_cost_time(us)(round_cost_time=35) [2024-04-22 16:23:27.315966] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.316048] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=49000, remain_us=617963, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.322756] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322841] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=82] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322871] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322895] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322916] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322932] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322947] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322964] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322980] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.322997] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.323032] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.323051] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.323067] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.323084] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.323099] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.323115] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [10413][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:23:27.337533] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [10412][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=83][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:385, tid:10477}]) [2024-04-22 16:23:27.341630] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=79] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:27.365191] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=24][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.365267] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=53][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.365293] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.365306] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.365330] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.368057] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4725] [2024-04-22 16:23:27.368096] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=36][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.368130] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.368161] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=27][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.368207] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=44][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.368224] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.368237] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.368294] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.368344] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.368363] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.368382] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=50, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.368414] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17] will sleep(sleep_us=50000, remain_us=565597, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.403903] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:193, local_retry_times:193, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.404070] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=132][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.404094] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.404106] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.404156] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.407744] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24][errcode=-4725] [2024-04-22 16:23:27.407794] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.407836] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=38][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.407861] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.407882] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.407902] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.407914] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.407965] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.408021] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.408043] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:27.408063] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=194, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.408129] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=48] will sleep(sleep_us=100000, remain_us=7419014, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=194, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.415039] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:27.415128] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=84][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774207260564935, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774207415022) [2024-04-22 16:23:27.415166] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774207215085, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:27.418636] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=95][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:50, local_retry_times:50, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.418717] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.418745] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.418779] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=32][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.418806] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.422701] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=51][errcode=-4725] [2024-04-22 16:23:27.422749] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.422782] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.422806] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.422822] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.422834] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.422845] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.422892] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.422930] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.422944] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.422980] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=34][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=51, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.423017] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19] will sleep(sleep_us=51000, remain_us=510994, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.474179] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:51, local_retry_times:51, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.474267] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.474296] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.474271] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF2A0-0-0] [lt=94] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=24, this=0x7f3cd2de56a0, extra_info="palf_id:1") [2024-04-22 16:23:27.474310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.474320] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF2A0-0-0] [lt=47] [PALF STAT LOG LIFE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=66882, this=0x7f3cd2de58d0, extra_info="palf_id:1") [2024-04-22 16:23:27.474323] INFO [PALF] read (log_cache.cpp:100) [10867][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=68] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=976, hit_cnt=8, read_cnt=8, hit rate=1.000000000000000000e+00) [2024-04-22 16:23:27.474359] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.474348] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [11497][T1_L0_G1][T1][YB42AC175881-000616AB0A3FF2A0-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.129:2882", total_slide_log_cnt=8, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=28, avg_log_submit_to_flush_time=5524, avg_log_submit_to_slide_time=66853) [2024-04-22 16:23:27.474492] INFO [LIB] stat (utility.h:1165) [10870][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=282] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=86, this=0x7f3cec8048f8, extra_info="") [2024-04-22 16:23:27.474530] INFO [LIB] stat (utility.h:1165) [10870][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=37] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=8, this=0x7f3cec804b28, extra_info="") [2024-04-22 16:23:27.478128] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:27.478192] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=60][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.478235] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=39][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.478261] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=22][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.478282] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.478299] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.478316] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.478374] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.478494] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=61][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.478522] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=24][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.478575] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=49][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=52, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.478564] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:23:27.478614] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=22] will sleep(sleep_us=52000, remain_us=455396, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.496556] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [10851][T1_ArcSender][T1][YB42AC175881-000616AB077FEBAD-0-0] [lt=39][errcode=-4018] try free send task failed(ret=-4018) [2024-04-22 16:23:27.508309] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:194, local_retry_times:194, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.508422] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=77][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.508496] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=71][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.508511] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.508537] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.512544] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:27.512598] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=51][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.512632] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=31][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.512679] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=43][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.512701] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.512718] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.512735] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.512803] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.512878] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=65][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.512901] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:27.512925] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=22][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=195, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.512975] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28] will sleep(sleep_us=100000, remain_us=7314168, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=195, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.530734] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=29][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:52, local_retry_times:52, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.530819] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.530841] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.530852] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.530878] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.534325] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=35][errcode=-4725] [2024-04-22 16:23:27.534368] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.534392] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=21][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.534407] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.534495] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=85][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.534510] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.534527] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.534576] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.534631] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.534662] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=28][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.534682] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=53, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.534716] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18] will sleep(sleep_us=53000, remain_us=399294, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.541974] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=66] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:27.549260] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=7255, clean_start_pos=4781502, clean_num=125829) [2024-04-22 16:23:27.587963] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:53, local_retry_times:53, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.588059] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=71][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.588087] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.588122] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=34][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.588159] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.591609] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:27.591659] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.591690] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.591713] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.591730] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.591742] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.591753] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.591800] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.591874] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.591891] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.591931] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=37][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=54, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.591997] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20] will sleep(sleep_us=54000, remain_us=342013, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.613132] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=51][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:195, local_retry_times:195, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.613215] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.613260] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=42][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.613273] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.613296] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.615274] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:27.615343] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774207457361382, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774207615261) [2024-04-22 16:23:27.615373] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774207415209, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:27.617336] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] [2024-04-22 16:23:27.617393] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=54][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.617435] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=39][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.617517] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=78][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.617549] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=28][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.617584] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.617604] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.617670] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.617768] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=89][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.617822] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=49][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:27.617867] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=196, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.617920] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=33] will sleep(sleep_us=100000, remain_us=7209223, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=196, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.646155] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=23][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:54, local_retry_times:54, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.646263] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=81][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.646290] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.646304] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.646339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.650277] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=24][errcode=-4725] [2024-04-22 16:23:27.650329] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=49][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.650371] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=38][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.650436] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=60][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.650522] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=82][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.650593] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=68][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.650610] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.650668] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.650730] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=52][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.650754] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=21][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.650781] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=55, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.650824] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=24] will sleep(sleep_us=55000, remain_us=283187, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.665759] INFO [LIB] stat (utility.h:1165) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3CAC-0-0] [lt=57] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=1, this=0x7f3cd2de5470, extra_info="") [2024-04-22 16:23:27.665844] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3CAC-0-0] [lt=54] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.129:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=854, avg_group_log_size=122) [2024-04-22 16:23:27.665895] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [11497][T1_L0_G1][T1][YB42AC175882-000616AB0B5C3CAC-0-0] [lt=46] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.129:2882", committed size=854) [2024-04-22 16:23:27.687792] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] Cache replace map node details(ret=0, replace_node_count=0, replace_time=9065, replace_start_pos=1258280, replace_num=62914) [2024-04-22 16:23:27.687830] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:23:27.698604] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [10825][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=39] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-22 16:23:27.705981] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:55, local_retry_times:55, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.706088] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=77][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.706115] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.706126] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.706153] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.709928] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:27.709969] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.710010] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.710036] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=24][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.710051] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.710062] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.710094] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.710142] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.710188] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=37][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.710203] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.710221] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=56, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.710258] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19] will sleep(sleep_us=56000, remain_us=223753, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.715397] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:23:27.718086] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=30][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:196, local_retry_times:196, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.718188] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.718217] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.718231] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.718256] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.721624] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=41][errcode=-4725] [2024-04-22 16:23:27.721662] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.721689] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.721705] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.721719] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.721758] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=37][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.721773] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.721820] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.721874] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.721895] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:27.721914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=197, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.721948] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19] will sleep(sleep_us=100000, remain_us=7105195, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=197, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.721892] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [10511][SysLocAsyncUp0][T0][YB42AC175881-000616AB0CEFFE3F-0-0] [lt=50][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713774207721790}) [2024-04-22 16:23:27.737220] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D6-0-0] [lt=233][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:27.737285] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D6-0-0] [lt=61][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:27.737333] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D6-0-0] [lt=45][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:27.737368] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D6-0-0] [lt=34][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:27.740648] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [10569][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:23:27.749993] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20728053760, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:27.766400] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:56, local_retry_times:56, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.766509] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=83][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.766537] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.766553] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.766601] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=33][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.769810] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=20][errcode=-4725] [2024-04-22 16:23:27.769844] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=32][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.769867] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=21][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.769885] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.769898] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.769906] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.769914] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.769961] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.770002] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=34][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.770018] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.770056] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=35][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=57, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.770097] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=23] will sleep(sleep_us=57000, remain_us=163914, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.815527] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=123][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:23:27.815592] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:27.815675] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774207661706940, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774207815510) [2024-04-22 16:23:27.815705] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774207615395, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:27.822118] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:197, local_retry_times:197, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.822218] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=68][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.822239] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.822250] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.822274] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.826244] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=19][errcode=-4725] [2024-04-22 16:23:27.826301] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=54][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.826336] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:23:27.826351] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.826366] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:23:27.826400] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=32][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:23:27.826412] WDIAG [SQL] open (ob_result_set.cpp:160) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.826499] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.826564] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=55][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:23:27.826594] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=25][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:23:27.826623] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=198, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.826669] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=27] will sleep(sleep_us=100000, remain_us=7000473, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=198, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.827268] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:57, local_retry_times:57, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:23:27.827355] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.827383] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:23:27.827403] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:23:27.827505] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:23:27.829773] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=28][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:27.829859] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.829921] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=58000, remain_us=104090, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713774207934009) [2024-04-22 16:23:27.831243] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [10808][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=132] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:23:27.887989] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:23:27.891944] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1][errcode=-4012] [2024-04-22 16:23:27.892018] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=71][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.130:2882") [2024-04-22 16:23:27.892051] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=30][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:23:27.892070] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:23:27.892086] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:23:27.892097] WDIAG [SQL] open_result (ob_result_set.cpp:211) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:23:27.892108] WDIAG [SQL] open (ob_result_set.cpp:160) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:27.892120] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:23:27.892132] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=10][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:23:27.892157] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=22][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:27.892200] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=39][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:23:27.892238] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=25][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:27.892275] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=34][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:23:27.892294] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=17][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:23:27.892319] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:23:27.892339] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1956704) [2024-04-22 16:23:27.892356] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=16][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:23:27.892372] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=13][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:27.892387] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:23:27.892399] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:27.892413] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:23:27.892478] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:23:27.892619] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=48][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:23:27.892651] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=29][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:27.892673] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=19][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:23:27.892687] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=12][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:27.892707] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=18][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:23:27.892722] WDIAG [SHARE.SCHEMA] operator() (ob_multi_version_schema_service.cpp:2425) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=14][errcode=-4012] fail to get baseline_schema_version(ret=-4012) [2024-04-22 16:23:27.892735] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2495) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=11] [REFRESH_SCHEMA] end refresh and add schema(ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], cost=1958739) [2024-04-22 16:23:27.892759] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:308) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=22][errcode=-4012] fail to refresh and add schema(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:23:27.892784] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:314) [10477][SerScheQueue1][T0][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=22] [REFRESH_SCHEMA] end refresh schema with new mode, (ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], used time=1958818) [2024-04-22 16:23:27.896368] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.896516] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] already timeout, do not need sleep(sleep_us=0, remain_us=1996296, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.898568] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.898649] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=1000, remain_us=1994164, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.901420] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.901567] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=2000, remain_us=1991246, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.906180] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.906259] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=3000, remain_us=1986553, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.912132] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.912167] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=4000, remain_us=1980645, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.918090] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.918125] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=5000, remain_us=1974686, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.925156] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.925188] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=6000, remain_us=1967623, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.926808] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=21][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:23:27.929609] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.929714] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=6897428, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=199, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:27.934041] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.934085] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=7000, remain_us=1958727, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.944226] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.944283] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1948529, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.950727] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:27.956069] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.956160] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=1936652, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.969775] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.969822] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=10000, remain_us=1922989, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.970803] INFO [LIB] stat (utility.h:1165) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=5674, this=0x7f3cd2def330, extra_info="palf_id:1") [2024-04-22 16:23:27.970866] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.129:2882", has_set_deleted:false}, accum_size=854) [2024-04-22 16:23:27.970939] INFO [LIB] stat (utility.h:1182) [10715][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=85, this=0x7f3ce8f43700, extra_info="") [2024-04-22 16:23:27.971046] INFO [LIB] stat (utility.h:1165) [10714][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=83] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=75, this=0x7f3cd2def560, extra_info="palf_id:1") [2024-04-22 16:23:27.983897] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.983983] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=11000, remain_us=1908830, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:27.998679] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:27.998732] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=12000, remain_us=1894080, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.014173] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.014220] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=1878592, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.015703] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:28.015741] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774207828972081, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774208015691) [2024-04-22 16:23:28.015756] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774207815725, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:28.031256] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.031334] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=14000, remain_us=1861478, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.033570] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.033658] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=6793484, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=200, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:28.049401] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.049519] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=1843294, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.068806] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.068913] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=16000, remain_us=1823900, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.088171] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=59] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:23:28.089233] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.089340] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=17000, remain_us=1803472, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.110912] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.110998] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=18000, remain_us=1781814, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.113684] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [10708][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=45] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:23:28.114239] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [10704][T1_Occam][T1][YB42AC175881-000616AB07FFED48-0-0] [lt=46] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:23:28.132926] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.133034] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=1759779, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.137517] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.137610] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=6689532, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=201, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:28.151480] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [10430][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=78] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20725956608, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:23:28.156145] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.156232] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=20000, remain_us=1736581, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.180122] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.180213] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=21000, remain_us=1712599, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.189496] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10597][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=49] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.189879] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10581][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.190089] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10575][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.190647] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10587][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=34] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.190687] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10591][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.191085] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10593][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=44] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.191426] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10574][RpcIO][T0][Y0-0000000000000000-0-0] [lt=43] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.191513] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10582][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.191925] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10570][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.192226] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10572][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.192506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10573][RpcIO][T0][Y0-0000000000000000-0-0] [lt=76] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.192658] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10592][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=43] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.192972] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10584][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.193232] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10583][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.193258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10571][RpcIO][T0][Y0-0000000000000000-0-0] [lt=130] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.193361] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10589][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.193472] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [10848][T1_LSArchiveMgr][T1][YB42AC175881-000616AB074FEBAD-0-0] [lt=58] gc stale ls task succ [2024-04-22 16:23:28.193511] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10585][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.193880] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [10543][Blacklist][T0][Y0-0000000000000000-0-0] [lt=43] blacklist_loop exec finished(cost_time=23, is_enabled=true, send_cnt=0) [2024-04-22 16:23:28.194140] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10595][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=61] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.194760] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10579][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.194805] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2024-04-22 16:23:28.194831] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=18] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=64) [2024-04-22 16:23:28.194865] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [10845][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=13] cannot_recycle_log_size statistics(cannot_recycle_log_size=369099, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:23:28.195058] INFO do_work (ob_rl_mgr.cpp:704) [10474][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=42] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:23:28.195337] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [10841][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=11104, mem_used=81110016) [2024-04-22 16:23:28.195341] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [10858][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=54] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:23:28.195602] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10578][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.197171] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [11013][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=50] current monitor number(seq_=-1) [2024-04-22 16:23:28.197310] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10588][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=47] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.197660] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [10576][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.198579] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [10590][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=108] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.199478] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [10580][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:23:28.203559] WDIAG load_file_to_string (utility.h:627) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:23:28.203583] WDIAG get_ethernet_speed (utility.cpp:580) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:23:28.203593] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:23:28.203604] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [10404][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:23:28.204251] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.204337] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=22000, remain_us=1688475, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.215893] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.129:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:23:28.215984] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713774207952292398, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713774208215877) [2024-04-22 16:23:28.216025] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [10842][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713774208015766, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:23:28.220521] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:23:28.220577] WDIAG [SERVER] do_heartbeat_event (ob_heartbeat.cpp:266) [10901][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] try reload schema failed(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=-4023) [2024-04-22 16:23:28.230121] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.230200] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=23000, remain_us=1662613, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.241147] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=9][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:28.241220] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=55] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:23:28.241248] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=24][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:23:28.241264] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=15][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:23:28.241298] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=32][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:23:28.241317] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:28.241358] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=38][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:28.241384] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=17][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:23:28.241409] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=22][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:23:28.241422] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:28.241470] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:23:28.241485] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=13][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:28.241497] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:23:28.241509] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:23:28.241521] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:23:28.241494] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.241546] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:28.241559] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:28.241574] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:28.241570] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10796][T1_FreInfoReloa][T1][YB42AC175881-000616AB080FEBCE-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=6585572, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=202, v.err_=-4725, timeout_timestamp=1713774214827141) [2024-04-22 16:23:28.241586] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:28.241611] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=22][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:23:28.241627] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:28.241640] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:28.241666] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:28.241692] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:28.241703] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:28.241713] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:28.241762] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10410][ConfigMgr][T1][YB42AC175881-000616AB0A8FED8F-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:28.241781] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:23:28.241797] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:28.241824] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=25][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:28.241844] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=18][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:23:28.241865] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=19][errcode=-5019] query failed(ret=-5019, conn=0x7f3cd45f0050, start=1713774208240947, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:23:28.241891] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=25][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:23:28.241911] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=17][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:23:28.242037] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=20][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:23:28.242058] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [10410][ConfigMgr][T0][YB42AC175881-000616AB0A8FED8F-0-0] [lt=19][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:23:28.242567] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D9-0-0] [lt=270][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:23:28.242624] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D9-0-0] [lt=53][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:28.242688] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D9-0-0] [lt=60][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:23:28.242720] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [10917][T1_L0_G2][T1][YB42AC175882-000616AB089C35D9-0-0] [lt=31][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:23:28.243181] INFO eloop_run (eloop.c:140) [10458][pnio1][T0][Y0-0000000000000000-0-0] [lt=59] PNIO [ratelimit] time: 1713774208243176, bytes: 66562393, bw: 0.027026 MB/s, add_ts: 1006862, add_bytes: 28533 [2024-04-22 16:23:28.257254] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.257338] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1635475, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.284493] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:23:28.284529] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [10477][SerScheQueue1][T1][YB42AC175881-000616AB0A6FEBF0-0-0] [lt=0] will sleep(sleep_us=25000, remain_us=1608283, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713774209892811) [2024-04-22 16:23:28.285171] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [10483][LSSysTblUp0][T0][YB42AC175881-000616AB0D2FEDE5-0-0] [lt=66][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713774194206222}) [2024-04-22 16:23:28.288348] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [10431][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:23:28.291322] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=1][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:28.291399] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:28.291477] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=72][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:28.291507] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=27][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:28.291532] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=22][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:28.291557] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=18][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:28.291569] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:28.291583] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:28.291595] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:28.291609] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=13][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:28.291625] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=15][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:28.291665] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=38][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:28.291683] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:28.291710] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=16][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:28.291739] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=21][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:28.291753] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:28.291763] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:28.291775] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:28.291789] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:28.291800] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:28.291825] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:28.291860] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=29][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:28.291871] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:28.291880] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:28.291900] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC7C-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:28.291916] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC7C-0-0] [lt=13][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:28.291931] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:28.291969] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:28.291988] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:28.292020] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:28.292040] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cc12b0050, start=1713774208289253, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:28.292142] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208289211), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:20:58.240169, ballot number:6', 'HAPPENED_TIME', '2024-04-22 16:20:58.277189', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:28.294088] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=36][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:23:28.294162] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=55] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:23:28.294195] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:23:28.294231] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=34][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:23:28.294247] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:23:28.294262] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:23:28.294272] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:23:28.294286] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:23:28.294301] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:23:28.294314] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=12][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:23:28.294328] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=13][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:23:28.294341] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=12][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:23:28.294354] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:23:28.294379] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=15][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:23:28.294395] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:28.294410] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:23:28.294479] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=67][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:23:28.294498] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=15][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882), ret=-5019) [2024-04-22 16:23:28.294518] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=17][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:23:28.294534] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=13][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:23:28.294561] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:23:28.294589] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:28.294604] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:23:28.294616] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=11][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:23:28.294659] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [10538][Occam][T1][YB42AC175881-000616AB085FEC7D-0-0] [lt=15][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:23:28.294691] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [10538][Occam][T0][YB42AC175881-000616AB085FEC7D-0-0] [lt=27][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:23:28.294712] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:28.294756] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:23:28.294771] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882), is_user_sql=false) [2024-04-22 16:23:28.294789] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)") [2024-04-22 16:23:28.294806] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f3cd1498050, start=1713774208292311, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882)) [2024-04-22 16:23:28.294888] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [10538][Occam][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713774208292268), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:20:58.277398', '', '', '172.23.88.129', 2882), ret=-5019)